Because the settlements there kept being destroyed in wars:
“In the days of the Hundred Kingdoms, many petty kings had claimed dominion over the river mouth, amongst them the Darklyn kings of Duskendale, the Masseys of Stonedance, and the river kings of old, be they Mudds, Fishers, Brackens, Blackwoods, or Hooks. Towers and forts had crowned the three hills at various times, only to be thrown down in one war or another. Now only broken stones and overgrown ruins remained to welcome the Targaryens.” (WOIAF)