Yeah, it's a self defeating cycle you've got for yourself. We stutterers do a lot of "mind reading", which is when we try to think what the other person must be thinking about us, our stuttering, how we come off, etc... Even though we may really feel like we know what they're thinking based on their reactions or expressions, we really dont. And it takes time and effort to slowly refrain from automatically mind reading. In my experience and the experience of many other people, no one really cares that you stutter. They have no idea how big of a deal stuttering is so to them it's just a little nuance about us. Our issues with our own speech trump how anyone else feels about it. So you're going to have to learn to just stutter and not worry what others may be thinking (you wont know anyway). Even if they do think it's weird, its better to stutter and take the chance than to worry, freeze up and have it be a much more awkward situation than it needs to be.