Am I sensitive for feeling this way?

Am I sensitive for feeling this way? I really have a problem with people who say "What?" in response to my stuttering. It's not so much the word but how they say it. I have encountered people who respond rudely to my stuttering, and it makes me very self conscious. I usually say "I am sorry; I did not catch that" in response to something I may have not heard correctly. Can anyone here relate?