It happens with older castles – your original motte-and-bailey castles were all wood. Stone castles are hella expensive, and take a long time to build, whereas wooden castles can be made from materials close at hand, they require less skilled labor to build, etc.
The Blackwoods I’m guessing initially built a wooden bailey and then built a stone wall around it later on when they were richer and could afford to.