There’s plenty of greenseers and wargs outside of high birth – you have the Green Men on the Isle of Faces, you have all of the wildling skinchangers (Orell, Varamyr, Haggon, Borroq, Briar, Grisella, etc.), there’s the Children of the Forest greenseers, there’s the Ghost of High Heart, and so on.
But in general, I think the reason that you don’t see evidence of the dead dreamers is that in a society with a high rate of child mortality, children who just die in their sleep aren’t unexpected, and if it’s a thousand across ~fifty years spread across an entire continent, it would be very hard to see the pattern.