I do the same thing, I try not to do this but it’s a pretty well engrained habit. However, I find I’m slightly more fluent when I maintain my eye contact instead of looking away.