I totally get what you mean, and it's okay to feel whatever comes naturally. I personally don't mind it too much, it honestly depends on how they say it. If it's just a simple "what?" as in they just didn't catch it, then yeah it hurts a little but I just repeat myself or spell it out. If they obviously sound annoyed then yes I get a little irritated at them