Whenever this happens to me I'll usually say something like: "I apologize, I have a stutter and get stuck on works that start with S-M so that's why I said 'little'." 9 times out of 10 they'll apologize or feel embarrassed. This then gives me the opportunity to reassure them that it's OK and that they had no way of knowing beforehand. You never know why a person might be feeling 'snippy' or impatient. Maybe they're having a bad day or just dealt with a rude customer. In my experience, being direct/honest/vulnerable about my stutter instead of trying to hide it due to shame – which is such a difficult thing for a disfluent speaker to do! – I've noticed that others are more likely to 'soften' and be understanding. Utilizing this approach has led to many moments of connection between myself and the person I'm talking to.