I'm in the same, exact boat. Contrary to your experience, it fucked my confidence, though. More than 80% of it is definitely anxiety and triggered hyperawareness of our speech. I went to plenty neurologists and psychologists. Had MRTs, examinations etc. and there doesn't seem to be anything noticeably wrong physiologically. I also have near zero stuttering around docs, especially psychologists. I had a few psychologists who worked with stutterers and they told me a purely genetic/physiological/neurological stutter isn't super sensitive to situations. It can be way less in some situations but it's never fully gone/unnoticeable. I still believe that there's some neurological part but it's rather mild and I have subconsciously learnt techniques to deal with it. But the main, disabling part is anxiety. I'm 100% certain. For example, if I'm with my friends I stutter a lot, and at some point I feel left out because I can't participate in the conversation the way I want to. However, if I'm drunk or had a significant positive experience just before meeting up with my friends, my stutter is almost 0. You should see a SLP (speech-language pathologist) and maybe talk to a psychologist to clear things up. If you're super paranoid like myself, see a neurologist as well.