We're affected by our experience in certain situations, if those situations repeat themselves we're more inclined to react to them and act in them the way we already did. Past experiences have a tendency to shape our future perfomances in the same environment. This is why, e.g, sometimes it's easier to act the way you want in a new situation than in your old and safe environment. We get used to our role there. I used to stutter severely while reading out loud in class but was fluent the same evening while reading my sister to sleep. Couldn't brake this "spell" all my school years. Stress is a big trigger but so is our inability to change the old patterns our mind works in in familiar situations. Anxiety can also be a consequence of our "inevitability of stuttering" mindset coming in. Maybe concentrating on where it gets bad for you isn't gonna be useful for now, maybe you need more positive outcome in other environments so that your mindset would get a confidence boost from somewhere else.