A lot of stutterers hate/dislike their stuttering or are ashamed of it. To them it might be like hearing a someone tell someone they like their tourettes. The idea is so outside of their mind they have a hard time comprehending why anyone would think that way. He was probably taken aback that you could compliment or like something that might cause him a lot of grief and pain.