It’s kind of complicated, and it ultimately comes down to the relationship between land and labor forces.
Unless you were literally a slave, and slaves did exist under feudalism although it had mostly died out by the 10-11th century, you didn’t spend 100% of your time working for the lord. Even the lowest serfs, villeins, cottagers, etc. only worked part of the week on lands held by the lord in demesne, and the rest of the time they would work on their own fields which they had been given a lease to as part of the feudal agreement.
You can think of this arrangement as a balance of the needs of the landowner and the needs of the workforce: the lord couldn’t and wouldn’t farm their entire manor themselves, and didn’t need the whole of the manor to provide food for their household and personal servants. At the same time, the number of workers who would be needed to farm the whole manor have to be fed and clothed and housed somehow.
Trying to hold the whole of the manor yourself would mean that you’d need to maintain and manage a large workforce of either slaves or wage workers, which would require large up-front ouflays (slaves have to be bought and then fed and clothed and housed sufficiently to prevent them all dying, wage workers have to be paid enough to buy those things themselves) and a lot of management to ensure that people who had no personal stake in the output of their labor would work more than the bare minimum to avoid beatings or firings (respectively).
And one of the things that the Middle Ages lacked was large amounts of capital and managerial capacity. It was much, much easier for a lord to lease out land to peasants who would then feed, clothe, and house themselves (thus freeing you from the cost of doing it yourself), then collect rent and taxes from them (without having to manage them yourself), while making sure you got enough free labor to do the work on the lands you kept for yourself.