I know that armies in the middle ages weren’t like big but why not? Who did fight in the army after all?

Armies tended to be small because medieval governments weren’t large or sophisticated enough to raise the revenue needed to pay and supply large armies, especially for long periods of time.