I have the same problem. My stutter (blocking, mainly) gets a lot more frequent with certain people, despite the fact that I’ve made huge progress with it since my childhood. Like with my older sibling who used to bully me (even though I am an adult now) and other people I feel are not patient or have an erratic or discomforting presence. I’m still working on it. Avoidance has been my only strategy, which only works for some situations, temporarily. But just know you aren’t alone. We’re doing our best and it’s a process.