I've had very much the same problem. It took me a lot of time fully realize and internalize that your stutter is not who you are. It's just a little problem in your speech. But by letting it influence other areas of your personality, like your confidence or feeling of self-worth you're empowering it. Because nobody really gives a fuck about it except yourself. The people who like you still like you when you stutter. The people who don't like wouldn't like you if you didn't. But even if they did, their opinions don't matter. The only thing that matters is the respect you have for yourself.