First, Tywin’s army was probably nowhere near 20,000 by the time of Tywin’s death. He’d taken casualties at the Green Fork, the retreat from the Green Fork, some casualties from raiding and counter-raiding, heavy casualties at the Fords, and some casualties at the Battle of Blackwater. I would estimate that of that 20,000 perhaps 12-15,000 were left.
Of those, most left when Tywin died.