Well, Garth Greenhand wasn’t really a ruler per se, more of a nature/fertility god who does seem to have been venerated across the whole of the Reach. It was his eldest son Garth the Gardener who was the first King in Highgarden, and based on what we know about subsequent Gardener Kings, yes his kingdom was small by later standards (keep in mind, power is relative; if everyone else is a petty king…).