I think Ned knew because Ashara told him, and his goal was to rescue his sister.
The Kingsguard were under orders to protect Rhaegar’s son and, after the Sack of King’s Landing, his heir. They fought to the death out of loyalty to House Targaryen.
Tearing down the tower probably would not have taken that long – undermine a load-bearing wall and you could have the thing down in a day or two.
The Daynes accepted Arthur’s death, I believe, because of the death of Ashara and Ned’s daughter.