No, I think what you're doing is good. You not breaking eye contact communicates to the stutterer that you're staying with them in the conversation. It communicates that your present and not perturbed. Which lessen the pressure to speak fluently and doesn't exacerbate their stutter. It's common for stutterer to break eye contact during a speech block because they're attention is now on their block and then trying to overcome it. They might also break eye contact because they struggle with feelings of shame or embarrassment due to their face contorting when they block. They haven't dealt with those feelings yet. So basically it's not you, it's them.