Suppose Maegor didn’t have to deal with the Faith militant and his nephews and nieces after seizing the throne. Would he have tried to invade Dorne?

Most definitely. Maegor would have needed a war to legitimize his reign and help to unify the kingdom after the rebellions during Aenys’ reign, and Dorne was the unfinished business. 

Gods show mercy on Dorne, because Maegor would not have had any.