Well, let’s ask Stannis:
“It was justice,” Stannis said. “A good act does not wash out the bad, nor a bad act the good. Each should have its own reward. You were a hero and a smuggler.” He glanced behind at Lord Florent and the others, rainbow knights and turncloaks, who were following at a distance. “These pardoned lords would do well to reflect on that. Good men and true will fight for Joffrey, wrongly believing him the true king. A northman might even say the same of Robb Stark. But these lords who flocked to my brother’s banners knew him for a usurper. They turned their backs on their rightful king for no better reason than dreams of power and glory, and I have marked them for what they are. Pardoned them, yes. Forgiven. But not forgotten.”
Stannis claims that he is the rightful king, as Joffrey is not Robert’s son, and he is Robert’s oldest brother. And he quite rightly points out that Renly has no legitimate claim to the Iron Throne and isn’t bothering to put forward one, so the lords who are trying to make Renly king have no excuse.