How big were actually garrison of a castle in Middle Ages!?

Anywhere from a few dozen to a few hundred men, rarely more than that, because the point of a castle garrison was to be able to outlast a full army in the field, and one of the ways you do that is having fewer men needing to be fed on the inside than on the outside.