Good question!
I think there are a couple different factors going on:
The Sistermen are not part of the mainland but rather out on the extreme periphery of the Vale:
- Because the Sisters became part of the Vale after the Battle of Seven Stars, they don’t have the cachet of having fought for the Andals and the True Religion (in the same way that a certain kind of WASP looks down on people who didn’t come over on the Mayflower or whose ancestors didn’t fight in the American Revolution) and are instead provincials.
- Because they live far off, they’re not a regular part of court life, and although definitely the highborn families would be treated as highborn, they certainly wouldn’t have the cachet of the inhabitants of the Vale proper.
- Because they don’t live on the mainland, they don’t share mainlander concerns about mountain clans raiding or growing seasons or mountain passes being closed off by snow, etc.
The Sistermen are moreover a historically rebellious and quasi-criminal part of the periphery.
- Before the Andals came, the Sistermen were pirates and slavers, neither activity likely to make them popular with the people they steal from or the people they steal. And to the present day, a big part of their economy is smuggling, and wrecking – the mainlanders despise honest merchants, so these guys are going to be seen as the lowest of the low.
- As the mainlanders see it, the Sistermen started the Worthless War by being greedy pirates who poked the wolf one too many times, and not a small part of the mainland suffered as a consequence.
- Then after the Arryns rescued the Sistermen, they rebelled against them during Aegon’s Conquest, and then against them and the Targaryens during the Blackfyre Rebellions.
And finally, there’s the whole inbreeding/webbed fingers and toes thing, which probably means they’ve interbred with the Squishers…