I have it similar to you, it comes and goes and anxiety has a big play in it however I still believe there is more to it than just emotion, something is still causing you to stutter, and enabling the emotion to cause a stutter. I ended up taking an mri a few years ago and they found scar tissue around the speech center of my brain. They couldn’t say that’s what the cause was for sure but I can only assume that’s what’s causing it for me.