The major factor was significantly higher mortality rates due to many people living in the same place with poor sanitation – no sewer systems, many unpaved streets, lots of domesticated animals – and poor public health. Even in the absence of major epidemics, people died of disease a lot.
Basically, if it wasn’t for the fact that people kept moving to the cities in large numbers, they wouldn’t have grown at all until really the 19th century.