I've been getting the "did you forget?" a lot recently. It absolutely sucks, no doubt, but you just gotta swallow it, politely correct them, and move on. Ultimately they mean well. The moment a stutter occurs is a vulnerable situation for you, no doubt, but it is also a vulnerable situation for those you are speaking with. You catch them off guard in a situation where they don't know what's going on. Sometimes I almost feel like it triggers a sort of fight or flight response. You can't really blame them for saying something that otherwise they would have realized was stupid. If you hit them with a polite "hey actually just so you know I have a speech impediment" they will be very apologetic and upset with themselves for suggesting such an absurd thing.