Then just say "did you not understand me". For me it's that the phrase isn't used in that sense, it's used when a statement is made and "did they stutter?" is used to emphasise that what was said was good. The phrase being used in this sense Implies that they didn't stutter and as a result, what they said had a point and/or wasn't stupid. If they did stutter then it would be the opposite. It's a very casual phrase that underlines how people really view a speech impediment