The short answer is that GRRM is more of a gardener than an architect, and his thinking about titles has changed over time, but without retconning to make everything consistent, you get anomalies.
However, keep in mind that lords can have more than one title. Wyman Manderly isn’t just the Lord of White Harbor, but also the Warden of the White Knife, Shield of the Faith, Defender of the Dispossessed, Lord Marshal of the Mander, Knight of the Order of the Green Hand, etc.
So it’s quite possible that a lot of these irregularities are due to the fact that these lords have both titles – one referring to the castle and one referring to the fiefdom – but one of them is more common.