I don’t think Stannis is making a factual statement as much as a philosophical statement.
The first thing he’s getting at is that, in their own realm, the king has no equals, and everyone around them is a subject connected to the king by chains of reciprocal obligation. As much as a king might want to just be pals with their old buddies, there’s both a basic power imbalance and an inescapable tinge of self-interest to their relationships with everyone around them. This is especially the case since proximity to the body of the King is the main route to power in feudal politics – everyone around the king, from the attendant who assists the king with the evacuation of his bowels and the cleaning thereof, to the person who refills the king’s drink, is looking to get something out of it, whether it’s royal favor or a chance to get bribed.
So a wise king has to look at the people around him with something of a jaundiced eye, constantly assessing them as to what they want out of him, what he’s getting out of them, and whether their political accounts are in balance, as opposed to trusting anyone. Because someone like Davos Seaworth who is so unconditionally loyal to the king that they would speak to the king’s best interest even when it goes against their own, is one in a million.

The second thing he’s getting at has to do with Stannis’ ideal of justice. Stannis believes very much that one of the highest duties of the king is to provide justice to all of his subjects:
“I shall bring justice to Westeros … Every man shall reap what he has sown, from the highest lord to the lowest gutter rat. And some will lose more than the tips off their fingers, I promise you. They have made my kingdom bleed, and I do not forget that.”
Thus, the king can’t have friends who get special treatment (privilege originally meant “private law”) but must treat all equally. And in order to achieve that state of impartiality, someone with Stannis’ beliefs would hold that there needs to be something of a remove, a distancing so that the law is predictable, consistent, and transparent.