is the manor the same as a castle?

No. A manor is an economic unit of agricultural production; a fiefdom consists of multiple manors. The manor is run by the manorial court, which is usually located in the manor house of the lord of the manor (begin to see why peasants weren’t always fond of the manorial courts?). 

A castle is a defensive fortification.