I always wondered if the wealth and power of House Frey isn’t a bit inconsistent. The crossing of the Green Fork just isn’t that important. Glancing at the map, the road to the North doesn’t pass through the Twins, the crossing would only matter for people going from Riverrun to the North, which is not a much used route. House Caswell, for example, holds Bitterbridge, where the Roseroad cross the Mander, a much more important road, shouldn’t they be way more rich and powerful than the Freys?

The road to the North doesn’t pass through the Twins but it passes right by the Twins (at least on the Lands of Westeros map), so if you’re taking the kingsroad, chances are the Freys are going to take their toll. 

In terms of the crossing, the Twins matter for anyone coming from or to Seagard, which is the Riverlands’ only west coast port. Given the proximity to White Harbor, there’s probably a good deal of passing trade that’s trying to get from west coast to east coast or vice versa without having to go the whole way around.