I think it's more that it reminds us of our own stutter, not that we're having a reaction that everyone without a stutter has. I tend to pick up on little details in the speech of others that informs me if they might have a stutter, but others (without a stutter) don't tend to. We're more familiar with the struggle, which is why we notice these things, I'd say.