How did the Lannister/Tyrell forces at the Battle of Duskendale manage to take heavy casualties given the massive advantages they were given?

Great question!

Sweeping victories in warfare tend to be rather rare, and even more rare without cost. Cannae is deservedly famous in military history as a complete victory, but even that total rout still cost Hannibal more than 10% of his army. 

For this very reason, the Byzantine manual of war (hat tip to @warsofasoiaf) advised against completely surrounding one’s enemy, because men who are completely surrounded will often fight like cornered rats and inflict surprisingly heavy casualties. Instead, the Strategikon advised that one should always leave a path of escape for the enemy, because then rather than standing their ground, the enemy was more likely to make a run for it, which would allow one to inflict heavy casualites on a fleeing enemy at a lower cost to oneself. 

Indeed, as military historians have pointed out, it was often the retreat from a battle that was the most deadly part of any premodern conflict, because a fleeing enemy couldn’t defend themselves against attacks coming from behind them, and often threw away heavy shields and armor in the process of flight. 

My guess is that Randyll Tarly and Gregor Clegane didn’t think about this.