Keep in mind that Kevan says that after the 2,000 men at Dragonstone have lost half their numbers, and Jaime’s forces have merged with Daven’s at Riverrun.
There’s no evidence of remobilization or re-routing, and Daven’s inability to actually control the siege is strongly suggestive that he doesn’t have more troops than the 2,000 that the Freys have.