You’d definitely expect to see a fair number of smallfolk in the area, working the lands attached to the castle – whether they’re clustered together in settlements or more spread out, harder to say.
To take a really rough rule of thumb, take the garrison and multiply by 300 to get the number of smallfolk supporting the castle.