I don’t think that’s accurate. There are a handful of ancient famous castles that are that old – Winterfell, Casterly Rock, the Eyrie, etc. – which happen to be very old, and we spend a lot of time in and around them in ASOIAF.
But there’s a lot of more recent castles. And I would imagine castle-building is kept up through maintenance, repairs, improvements, etc.