That is a big part of why I take this position: the Daynes have absolutely zero reason to give a shit about Brandon Stark – a betrothed man who was never going to take care of Ashara when she got pregnant, a man they likely never met given the timeline of his capture vis-a-vis the Tourney of Harrenhal.
But Ned’s story is undeniably tragic – star-crossed lovers divided by civil war; True Love vs. Family, Duty, Honor; untimely death caused by miscommunication, etc. So you have a sense of why the Daynes would care, and why Ned would freak out in AGOT when he hears Ashara’s name – remember, she doesn’t threaten R+L=J. If anything, Ashara’s a good smokescreen to distract people from the truth. But Ned gets uncharacteristically emotional about it, I think because it hits on a different sore spot.
And I agree, there’s almost no way that Ned wouldn’t have seen a parallel between his dead daughter and his “bastard son” coming into and out of the world so closely together.