In Catelyn II of AGOT they talk about Brandon being a shadow between them in their marriage, there’s the way that she describes Brandon in Catelyn VII of AGOT.
And in ASOS, she describes her first impression of Ned:
“She remembered her own childish disappointment, the first time she had laid eyes on Eddard Stark. She had pictured him as a younger version of his brother Brandon, but that was wrong. Ned was shorter and plainer of face, and so somber. He spoke courteously enough, but beneath the words she sensed a coolness that was all at odds with Brandon, whose mirths had been as wild as his rages. Even when he took her maidenhood, their love had more of duty to it than of passion.”