Because you don’t stay wealthy by supporting moochers in a lavish lifestyle.
This being said is it unusual the Freys remained so wealthy given Walder’s numerous sons?
A whole bunch of Walder’s brood have probably “earned out.” Emmon Frey married the only daughter of Tytos Lannister, and you can’t tell me she wasn’t handsomly dowered, for example.
It is indeed, although partly Walder achieves that by being a cheap bastard.