Stuttering is an invisible "disability" - people think we're just nervous. In most cases (I say most, there are assholes) someone would never tell someone in a wheelchair "you need to work on your talking skills" because they KNOW they have a disability. Don't assume people know you stutter, you have to make it known. I generally introduce myself with a "Hey, I'm JoeOutrage. I have a speech impediment that causes me to stutter, so in case that happens I just ask for your patience!" This is generally beneficial in two ways: 1) People don't think you're nervous, and 2) You're generally a little more comfortable speaking in front of them. The stutterer themselves is usually a stutterer's worst enemy, so educate instead of retreating inward. If they still say things like that after educating, then they're an asshole and you cut them out of your life.