Some early researchers called stuttering what you do to avoid stuttering. I think OP has an overall point, however. At it's base, stuttering is just the disruption of speech. There is a reaction to that disruption (which in many cases is more salient to the speaker than the actual disfluency). And this is OP's point - we all react differently to the moment of stuttering.