It’s got to be smaller than the population of the wildlings, since the wildlings had a much, much bigger territory to operate in which included land far more amenable to human habitation than mountains.
According to the Wiki of Ice and Fire, which cites Catelyn VI of AGOT, although I can’t find the citation’s source at the moment, the mountain clans can field 3,000 men. Taking the wildlings as an example, this would suggest that there are 10,000 people in the mountain clans in total, and a third of them can fight.