It's extremely common, a "secondary behavior". Eye contact is a lot of processing. Your brain would rather use that processing for something else, like thinking of words to say or concentrating on rate and timing. Maybe you could focus on people's chins or shoulders.