Absolutely, that’s one of the more realistic things about ASOIAF. Here’s the crucial factor to consider about the economics of the nobility – in feudalism, rents are generally fixed at traditional rates. Which means that the nobility are more exposed than most to economic shifts, especially shifts in prices.
One of the reasons why we see peasant revolts in the 14th century following the Black Death (which greatly decreased the labor supply and thus raised wages, at a time when noble incomes were declining because their rent-paying tenants were dying or running away) and then again after the Great Price Revolution (which raised the price of everything, and thus was a major real income cut for people on fixed incomes) is that these events hammered the economic position of the nobility, the nobility responded by trying to violently restore the balance of power (both by trying to freeze wages and worker mobility, which often meant attempts at enforcing or re-establishing serfdom), and the peasantry responded with violence in return.
Now, the greater nobles were better able to adapt to changing economic circumstances – they had more land and more liquid capital, so they could convert more easily to pasturage and thus get into the lucrative cloth trade, they could invest in new commercial and industrial ventures, etc.
But the lower nobles didn’t. Hence the figure of the impoverished nobleman, who becomes ubiquitous from Don Quixote to Jane Austen to the freaking Bluths.