Has Littlefinger some sort of agreement with Varys about not ratting him out? Telling the lie about the dagger with Varys right there seems to be insanely dumb otherwise. He must have been somewhat sure Varys would not report this, because he would have been in deep shit.

No he doesn’t, and yes it is insanely dumb. But then again, LF is a man who believes:

Littlefinger smiled. “Leave Lord Varys to me, sweet lady. If you will permit me a small obscenity—and where better for it than here—I hold the man’s balls in the palm of my hand.” He cupped his fingers, smiling. “Or would, if he were a man, or had any balls.”

This is why I consider LF to be too reckless to be the perfect conspirator.