Tyrion mentions that the Westerlings had sold off a large portion of their lands. How exactly would such a transaction take place in a feudal economy? Would there be restrictions on who they could sell to and for how much?

Discussed somewhat here. 

The Westerlings selling their land is a highly unusual event in Westeros – the only other times we hear about selling land is in the context of the Tarbecks forcing people to sell their land through threat of armed force, so voluntary (to the extent that the necessities of poverty qualify as voluntary) land sales are a sign that the feudal order is in crisis. 

It suggests that the Westerlings were falling into genteel poverty, such that their rental income had fallen massively behind their ability to service their debt, and that they were having to surrender the collateral they had put up to secure the loan. 

Legally, this could be quite tricky. In Medieval England, for example, the feudal principle of “Nulle terre sans seigneur” (no land without a lord) meant that selling land outright, known as “alienation of lands by will,” was actually legally impossible until the late 12th century. (The Magna Carta, for example, says that “No free man shall henceforth give or sell so much of his land as that out of the residue he may not sufficiently do to the lord of the fee the service which pertains to that fee.”) Selling land was legalized by the Statute of Quia Emptores in 1290, although the buyer was “required to assume all tax and feudal obligations of the original tenant,” so the land remained under the same lord as before. It wasn’t until the Tenures Abolition Act of 1660 that those feudal obligations were eliminated.