Here’s how the WOIAF puts it:
“the Barrow Kings to their south, who styled themselves the Kings of the First Men and claimed supremacy over all First Men everywhere, even the Starks themselves.”
The use of the terms “styled” and “claimed” suggests their reach was larger than their grasp, that they held authority over the south-west of the North but not the whole.
So more like the Fisher Kings or the Yronwoods.