It’s more the latter, plus the nobility are going to be really really unhappy with the idea that their future king broke an oath of betrothal. Kind of a big sign that he’s neither stable nor trustworthy.
And Aegon V could have disinherited Duncan and gone with another son, so he couldn’t have held out. Indeed, his abdication was sort of a way to force a resolution of the scenario by doing that voluntarily and removing any potential question of the legitimacy of his younger brother’s claim.