Hunting lodges were seasonal dwellings, meant to hold the lord and his coterie during the hunting season for whatever animal was in good supply (deer, boar, birds, etc.).
Manors were more commonly the permanent dwellings of the nobility than castles. Castles were expensive military fortifications, required royal license, and usually not well-suited for civilian use, although one has to make a significant exception for the castles raised by the upper nobility who could afford to splash out for deluxe castles that could do both.