How is it some lines of noble families end up as poor as the Tolletts, or end up becoming merchants like the Gulltown Arryns?

Good question!

There are many ways a noble family can either fall into genteel poverty or experience downward social mobility into the merchant class or even below (just look at the Heddles), but most of them come down to the relationship between rent, income, and debt:

“Annual income twenty pounds, annual expenditure nineteen pounds nineteen and six, result happiness. Annual income twenty pounds, annual expenditure twenty pounds nought and six, result misery.”  (Dickens)

Almost by definition, the major source of income of a noble family is rent income from their lands, and rents were overwhelmingly set by custom and tradition. This meant that most nobles were living on something like a fixed income, which meant they were very vulnerable to changes in prices. Crop failures, rebellious peasants demanding wage increases, competition from foreign countries, all of these things could seriously negatively affect the bottom line. 

This could be especially problematic, because nobles were supposed to A. live an ostentatious lifestyle (hunting, hawking, entertainment, and fancy clothes cost a lot), and B. not care about money like some grubby bourgeoisie. A combination of these two social expectations means that a lot of nobles went into debt to keep up with their social peers, and since land was the only collateral they had…you can see where this goes.

So you get a gradual process by which trying to keep up with one’s station and the Joneses lands you in debt, the debt eventually gets larger than your ability to pay, you end up losing bits of your land to satisfy your creditors, that reduces your income and exacerbates the problem, and so on…