Tommen was born 2 years after the Greyjoy Rebellion & he was “sucking at the Lannister woman’s teat” the last time Ned saw him. There’s a great post on r/asoiaf titled “Jorah Mormont, the reason Ned & Robert never kept in touch?” which explains that whilst Ned & Cat were visiting in KL & waiting for Robert to return, news of Jorah’s slaving came through first & Ned rushed north, hence not seeing Robert since the Lord Hypocrite of Pyke rebelled.