WOIAF explains that the Order was founded very early in the Dawn Age, before the Pact. Uther of the HIgh Tower, the founder of House Hightower, had two sons: Peremore, who gathered together the scholars, and Urrigon, who bequeathed them the land which the Citadel would be built on.
Which is weird as hell, because it means the Citadel is more than 10,000 years old, making it easily the oldest institution in the world, and what’s more confusing, means it predates the arrival of writing in Westeros by 4,000 years. Or so the Citadel tells us…