I think it ultimately comes down to that Valyria was weaker. Smaller territory, fewer soldiers, etc. With the Rhoynar, Valyria could hit them with an entire empire, such overwhelming force that it wasn’t a defeat with a peace treaty and some loss of territory, but a total civilizational collapse.