It's not particularly offensive, but I don't like it either. I don't get why it's even a saying, given the fact that stutterers are so uncommon, and that I honestly never hear anyone else stutter. It's just another way of asking "Did you not understand me?", but I find it dumb