It’s probably just due to shame. You assume your listener will have a horrified expression (or mocking) when they hear you speak. So, by looking away, you are “confirming” you are correct. But by looking at the listener, you break the spell, tricking your brain back into confidence.