So I take it that most smallfolk aren’t privy to the fact that Targaryens tend to be madmen.

It’s popular knowledge – rumors going around that Rhaegel liked to dance naked through the Red Keep, etc. But I think it’s more that they don’t really care unless it has negative impacts on them, as with Maegor.