Sure, it can be disconcerting to hear someone stutter. I think we stutterers also tend to notice it more. But getting outright angry doesn't sound quite healthy. Maybe bring it up to your speech therapist?