could the riverlands have eliminated its massive border problem by trying to win the great game and unite its neighboors under its banner? or did it simply not have the potential might necessary to do so in any circumstance?

The Riverlands certainly tried…we know they invaded the Reach following Gyles III’s invasion of the Stormlands, for example. 

And I wouldn’t be surprised if the downfall of more than one Riverlands dynasty was in overreaching themselves trying to conquer the northern Crownlands or the northern plains area of the Reach or trying to push from Saltpans to grab Wickenden or perhaps even trying to push north past the Neck.