Just because you haven't seen the phrase used in its intended use or that it doesn't bother you, doesn't mean it doesn't happen to others or doesn't bother anyone else. Even if it's not massively so, it's casual. I know it's not an overt phrase but it has underlying issues that people continue to sweep under the rug. Stuttering is still a disability that society seems to think ok to mock or to be afraid of. Any sort of casual ableism, or casual ableism speech shouldn't be used. If this phrase doesn't worry you, then don't worry about it. But again, just because it doesn't bother you doesn't mean it doesnt bothers others