when introducing a castle/fortress in a story, is it a good idea to introduce the layout of the structure in one chapter, and then flesh it out in the following chapters, or should you be vague when you introduce it, and thus keep a sense of mystery as you explore it over the course of a story?

Depends on the story. 

The Gormenghast series starts with an exploration of hte layout of the structure of the castle of Gormenghast, because the isolating and paranoid atmosphere of the castle, the way in which hyper-stylized Victorian ritual and Gothic decay go hand in hand, is key to the book’s themes (nature vs. nurture, stasis vs. destructive change) and its plot: when characters are threatened, it’s by being locked in rooms and having to clamber across cathedral-like rooftops looking for an entrance, the dramatic fire in the library drives characters mad, duels take place on swaying, rotten beams over abandoned ballrooms, etc. 

So starting with the layout of the castle works there, because the setting is so integral to the story.

However, if you want the castle to be mysterious, you probably shouldn’t start with a detailed explanation of every nook and cranny. Be vague, but I would also recommend using the limitations of a character’s POV. Thus, the reader can only see as much of the castle as the character can, and only gradually as the character moves through the castle do they learn about it.