Everyone is different whenever it comes to what they prefer the listener to do while they are speaking. I prefer the listener to look me in the eyes and not look away as when they look away I feel less engaged, flustered, and embarrassed. As long as you are listening and responding in a manner as you would to a person who does not stutter, you are not being rude!