Gulltown is the primary trade port of the Vale, but I don’t know if I’d agree that it’s “the crutial link between Braavos/the North and southern Westeros.” Nothing’s stopping Braavos or ships from White Harbor from sailing directly to the Trident or King’s Landing or the like.
As for why the Graftons are not particularly prominent, I think part of that is relatively recent – they backed the Targaryens during the Rebellion, forcing Jon Arryn, Ned Stark, and Robert Baratheon to take Gulltown by storm. They no doubt suffered reprisals from that.
The other part is that they’re not the only game in town. There’s the Arryns of Gulltown, the Shetts of Gulltown and Gull’s Tower (who are vassals to the nearby Royces of Runestone), etc.