It’s something I associate more with the Early Modern period and the Enlightenment, but snobbery by the old blood against the nouveau noblesse as it were was extremely high. Big part of the reason why Anne Boleyn fell was that the old nobility didn’t like the idea of a knight’s daughter getting above her station; in the ancien regime, the noblesse d’épée despised the noblesse de robe as a bunch of upjumped bourgeois social climbers.
As for why people despise the Freys, it’s a mixture of the fact that the Freys are ambitious and obsequious social climbers, that they’re grasping middlemen who are “toll collectors” and “coin clinkers” (i.e, their money comes from trade not land), there’s the subtext that they enforce their monopoly by burning other people’s bridges and that they tried more than once to conquer the Neck (hence their hatred of the crannogmen), and that they’re disloyal (the Blackfyre Rebellions, Robert’s Rebellion, the War of Five Kings), etc.