That’s an excellent question, because I realized that I was also unsure about this question. My asearchoficeandfire-fu was failing me, but @goodqueenaly found this rather interesting passage from Tyrion III of ASOS:
“My lords, with your leave, I propose to travel to the Vale and there woo and win Lady Lysa Arryn. Once I am her consort, I shall deliver you the Vale of Arryn without a drop of blood being spilled.”
Lord Rowan looked doubtful. “Would Lady Lysa have you?”
“She’s had me a few times before, Lord Mathis, and voiced no complaints.”
It’s a rather surprising moment, because I had been looking in AGOT for where Tyrion discusses Littlefinger’s boasting at court, and had had a false memory of him boasting about both Tully girls when in fact the text only mentions Catelyn.
So this raises some interesting possibilities: either Littlefinger had realized what happened at Riverrun, but was unwilling to admit the full truth either to himself or anyone else, or he and Lysa had actually had an affair either in the Vale or King’s Landing (which is something that’s been debated but never proven), or Littlefinger was inadvertently telling the truth while lying.