Your theory doesn’t sound crazy at all to me…

While the clothing might suggest a link to the Green Men of the Isle of Faces, the fact that Garth Greenhand didn’t get along with the children suggests otherwise.
My crazy theory: I think Garth was a First Man shaman/king who had access to a different kind of magic than the Children, more of a flora/wood-bender to the Children’s fauna/earth/water-bending. Hence why he was considered a god.