I think it’s very insensitive, but not offensive. Aside from 5 people, I haven’t met any other stutterers. It’s not common enough that it would register in people’s minds, so I guess I cut them a little slack.