I do the same. I just try to force myself to at least glance at the person when I notice myself doing it. Something that's important to realise though, is that it's normal not to make constant eye contact when talking. So don't worry too much about it.