In AFFC Sam V it says that “Randyll Tarly was known in Old Town, but little loved” and Sam is worried about being recognized by “one of his lord father’s enemies”. Do we know/can we surmise the backstory to this or should we just attribute it to Randyll Tarly just being an unlikeable dickhead?

A combination of geopolitical rivalries and personality. The Tarlys are Marcher Lords of the Western Marches, and may well be the most prominent of the Western Marcher Lords, since the Carons of Nightsong owe their allegiance to Storm’s End. At the same time, the lands of House Hightower border the Western Marches, and there are some marcher lords like the Mullendores of Uplands, who are vassals of Oldtown. 

I would imagine there’s been some conflict over spheres of influence between Horn Hill and Oldtown.