What would happen exactly if all the targaryen male heirs died? Like in a boating accident, leaving Rhaella and Rhaenys as the last living members of their house?

You mean a King Ralph scenario? 

Depends on the politics of the situation, but the rules of Targaryen succession would say that the nearest male relative would inherit, which would have been Robert Baratheon.