I don’t think regicide is quite the issue – after all, Kingsguard have defended queens accused of adultery and therefore treason – as much as who Tyrion is. Yes, he is related to the royal family, but he is not of the immediate royal family and is on the queen’s side of the family to boot.
Hence, Tyrion isn’t covered by the oath of the Kingsguard to serve as bodyguards for the royal family so can’t call upon them to serve as champions in a trial by combat.