I think this is pretty common for stutterers. At least for me, it's definitely the case. The people I know well are already aware that I stutter and don't even react to it. There is zero pressure, no expectations for a fluent conversation from friends/family. When I'm around new people, I'm both trying to avoid and simultaneously anticipating that moment when a stutter occurs and the person I'm interacting with realizes it. It's distracting, for one. I'm unable to focus on breathing an my other techniques, and tend to default to unhealthy avoidance behavior (substituting words, using a lot of 'uh, like, um' transitions, etc). This can be especially worse if the person I'm talking to knows I stutter, but has never been particularly nice/supportive/patient about it. Of course, it's easier to tell oneself to stop caring than it is to actually do it. I'm hesitant to give any advice actually, as it's still something I struggle with, but I've found having someone with me who I am close to helps a lot in those situations. I can kind of feel like I'm talking more to them than the other person.