Imo, no. More often than not, the people who say that are the people that don't know what a stutter is/that a stutter is a disability and not just stumbling on your words (which is normal). They aren't intending to be offensive, If they knew that a stutter was a disability, I don't think they would use the phrase.