It was probably built, but I would guess that Bloodraven had it destroyed to prevent it from becoming a shrine to the Blackfyre cause.
As for the land/castle issue, Daemon “granted Daemon a tract of land near the Blackwater, with the right to raise a castle.” A castle was an improvement to one’s land, and at various times kings encouraged castle-building among their subjects (Edward I, for example) and at other times tightly controlled the licensing of castle construction and would pull down illicit fortifications (Henry VII).
And yes, prior to raising his banners, Daemon was a landed knight.