I have a different theory about the origin of the wildlings, and I think it matches the text better: selective immigration. You don’t like living under an Umber, you move North. You think it’s wrong for people in the same village to marry, you move North. You still want to sacrifice people to the trees, you move North. After the Others fade from living memory, the prospect seems far less daunting.