Same problem. I'm usually fluent too , i don't stutter much and when i do i'm pretty good at hiding my blocks with "uhms" and such, but anxiety is an hard enemy to overcome...self-esteem issues too. I think sometimes it's more frustrating for us stutterers who have it mild, because we know we can do it if we try, and when we are confident and encounter a block anyway, it's very frustrating. The best solution in this case is to just keep talking to them; just throw yourself in situations you usually don't like, and you feel uncomfortable in. As Giovanni here said too, it's the correct way to boost your confidence and improve your stutter at the same time. I know it's not easy at all but it's the best way. I'm currently trying to overcome my anxiety issues and my problems with stuttering in general, hope it works out for you as well!