To give a real-world example, someone from Texas who goes to South America or Europe might be called a “Yankee” because that label has come to stand for all Americans, even though that Texan might object fiercely because, in their native Texas, that term has a different meaning and context that most people in South America or Europe don’t know. Likewise, an American might call someone with a Scottish or Irish accent “English” because their context for what those terms mean comes entirely from Hollywood and television, with no understanding of the historical background that makes those categories quite separate in their own context.
So in the context of Planetos, Westeros was conquered by the Andals 6,000 years ago, and most of the continent is Andal in heritage. Since the Andals came from Essos, to the extent that most Essosi know anything about Westeros, they know it as the land that the Andals conquered, lacking the context of the North’s continued independence, etc. Hence most people call everyone from Westeros an Andal, even when they’re actually a First Man.