Money, primarily. It’s not like the North didn’t use the mouth of the White Knife – the Wolf’s Den was fortified because that territory was useful to them. But the Manderlys had a depth of liquid capital that the less-fertile North simple didn’t have to hand, and because it was the only option, they invested all of it into a single location. This created the necessary labor demand for a proper city.
But yes, the development of the Wolf’s Den was likely inhibited by the castle being the Vale’s focus of attack during the Worthless War and then suffering the effects of civil war and repeated change in management.