Because I am seeing this described entirely incorrectly. It is NOT the case that the White Walkers didn’t harm wildlings – hell, when we see the Army of the Dead described in Sam I, they’re mostly wildlings – only that they didn’t attack them en masse. To quote Tormund:

“They never came in force, if that’s your meaning, but they were with us all the same, nibbling at our edges. We lost more outriders than I care to think about, and it was worth your life to fall behind or wander off. Every nightfall we’d ring our camps with fire. They don’t like fire much, and no mistake. When the snows came, though … snow and sleet and freezing rain, it’s bloody hard to find dry wood or get your kindling lit, and the cold … some nights our fires just seemed to shrivel up and die. Nights like that, you always find some dead come the morning. ‘Less they find you first. The night that Torwynd … my boy, he …’ Tormund turned his face away.