Minority opinion time!
Ned.
I don’t think Ned/Ashara detracts from Ned/Catelyn in the slightest. By contrast, I think it adds, giving another layer of commonality between the two. Ned/Ashara means that Ned and Catelyn were both disappointed in their match and both overcame it. I don’t put much weight on Ned not thinking about Ashara in AGoT; unlike ongoing issues with Lyanna and Jon Snow, wounds that reopened over the course of AGoT, Ashara’s role in Ned’s life is, tragically, resolved. By the time we get in Ned’s head, he’s in love with Catelyn, happy with his life with Catelyn, and has been for many years.
Nor do I think it detracts from any ongoing themes regarding Ned’s honour; this is specifically addressed in the text.
“When Ned met this Dornish lady, his brother Brandon was still alive, and it was him betrothed to Lady Catelyn, so there’s no stain on your father’s honour. […] Spring had come, or so they thought, and neither one of them was pledged.”
– Harwin to Arya, Arya VIII, ASoS
Whatsoever dishonour existed in an actual affair between the two (minimal) would have been fixed with a betrothal (entirely possible at the time). The important point of dramatic irony re: Ned’s honour and his sexual relationships is that he never cheated on Catelyn, not that he never loved another woman full stop.
I also think Ned/Ashara explains a few backstory elements more convincingly.
First, Ned’s atypical reaction to Catelyn’s mention of Ashara (Catelyn II, AGoT). Implied to take place fairly early in their marriage, Ned actually scared Catelyn. While not okay at any point, this would be another degree of overreaction altogether if Ned had only been crushing on Ashara. And despite the mention of Jon Snow, we know that Ned was reacting to the mention of Ashara as well, because he cracked down on gossip about her afterwards.
Second, how Ned discovered Lyanna’s location at the end of Robert’s Rebellion. Sure, maybe for the memory of the now very dead Brandon, Ashara betrayed her living brother…or maybe she told her very much alive star-crossed lover where her brother and his sister were. I like the latter explanation because it cuts out the middleman. It’s simpler and more compelling to me.
Third, the connections between Ned and House Dayne. Ned returned Dawn to them, no grudge was held over Ned killing Arthur, it seems likely they cooperated in hiding Jon’s parentage, the heir to House Dayne is Edric “Ned” Dayne, too similar to be a coincidence… Again, I guess it’s possible that they’d do all this for the brother of Ashara’s dead lover. And again, I think it’s simpler and more compelling that the Daynes cooperated with Ned because he was Ashara’s lover (free and willing to marry her at the time), and they understood that everything fell apart due to circumstances beyond his control.
Brandon/Ashara puts the actions of Ned and the Daynes throughout the Rebellion at one remove, to my mind. Possible, deliberately ambiguous in the text thus far, but I do prefer the more immediate connection as a motivation.
I don’t think we have enough information about Ashara to say what would have become of her had she lived, beyond that she would have had a lot of grieving to do. A peaceful life in Dorne away from the politics that caused all that grief sounds good, though.
^ Cosigned to all of this.