I don’t think he was loyal to Viserys in the slightest, only that at various times he saw more personal advantage inside the regime than outside of it.
And I think it’s kind of bizarre you’d call him loyal to Rhaenrya and Nettles, given that he was married to the first one while schtupping the second.