Was Robert really much better than Aegon IV? (Mistresses, bastards, disrespecting his wife and women, filling the court with syncophants, etc)

Absolutely. 

• Mistresses and bastards: Robert never let his sexual desires interfere with politics. For the most part, he left the womenfolk of the nobility alone, and even with Edric Storm there was never any suggestion that he would be legitimized and added to the line of succession. He never redistributed any land or royal offices as a result of his affairs. By historical standards (all those dragonseeds had to come from somewhere…) he was pretty good on this score.
• Disrespecting his wife: Robert was awful to his wife on a personal level, no question. But I wouldn’t say that he was as bad as Aegon IV or Aerys II or Maegor. And on a political level, Robert made good on the dynastic alliance – arguably too good. 
• Filling the court with sycophants: Here, I think it’s easy to paint with too broad a brush. Varys, Pycelle, Littlefinger, the Lannister hangers-on, definite mistakes in hindsight. However, most of these were either the result of completely hidden outside agendas that no monarch could have seen coming and/or carryovers from the previous administration. And against that, you have to balance the able governance of Jon Arryn, Stannis’ excellent tenure as Master of Ships, Renly’s political skills, and making Ser Barristan Selmy the Lord Commander of the Kingsguard. 

Add onto that that Robert completely avoided Aegon IV’s military failures, his wanton and brutal executions, his random and extra-legal seizure of his vassals’ lands, etc. He might not have done much as king, but as Terry Pratchett once said, what he mostly didn’t do was a lot of harm.