As with the first Red Wedding, the Freys aren’t going to have all 2,000 of their men inside the castle. The common soldiers camp outside while the nobility goes inside.
The Brotherhood Without Banners sneaks inside using the water gate, bar the doors, and kills everyone inside.