Try to treat him as you would anyone else and not think about it too much. My favorite listeners are those who are respectful and patient but we're otherwise completely indifferent to my stutter. They didn't care one way or another.