Why did ned brought so few men to the tower of joy?

Because the war was over, and he wanted to move fast. 

Also, if as I think, Ashara Dayne had spilled some of the beans to him, he might have suspected that he didn’t want a pro-Robert Baratheon army around him when he found Rhaegar’s son.