Why do you think the Blackfish never became a member of Robert’s kingsguard? Because Brynden himself didn’t want it, because he was too old or for some other reason?

Robert’s lack of attention to his own kingship, probably. Brynden is a universally-admired knight who’s not going to inherit, a white cloak would be a good consolation prize for Hoster, and he really wouldn’t mind the “no wives” thing.