Well, let’s start with in real life. Especially in the early Middle Ages, you would be widely seen as unmanly and not really living up to your obligations, especially if you were that blunt about it instead of saying instead that you couldn’t be there personally because you needed to guard against border raiders or to deal with bandits or you were ill or something. But by the 12th century, when the institution of scutage was developed, you’d just be paying your taxes as a good subject ought, so the social consequences change enormously over time.
In Westeros,look at how people think about Walder Frey as a coward and disloyal, etc. Unless there was some extenuating reason – no one else to look after the holdfast, for example – you would face a pretty sharp loss of face.
However, attitudes are different depending on other social norms. If you are a younger son like Leobald Tallhart, as long as your older brother is representing the family on the field and you’re needed to look after Torrhen’s Square, you’re not going to be treated the same way. If you’re the heir and your father is still of age, you might be left behind like Benfred Tallhart as opposed to being taken along like all of the Karstark lads, although Benfred forming the Wild Hares shows a certain impatience and a lesser but extant degree of social pressure.