I have a stutter and you would come across as an asshole to me tbh. When you quickly explained yourself I would understand and be less irritated but that initial reaction would absolutely happen. Also just as a sidenote, it might be worth it when thinking about your reactions to examine why you assume everyone with a stutter feels pain and embarrassment when they speak. TLDR: I’d try and tone that reaction from a laugh to a smile