I don't stutter when I'm alone, what's up with that?

I don't stutter when I'm alone, what's up with that? I never stuttered as a child but I grew up with a big brother and a dad who stuttered their whole lives. My brothers speech therapist warned my mom that because I was a baby I'd mimic my brother's speech but it was never really a big problem growing up. In the last 2 years it feels like my stuttering has just picked up a whole few notches which is odd because it wasn't there through my teens but now as a young man it's coming through. I work in customer and I can't catch a break lol. I suffer from anxiety and low self esteem and this very much plays the biggest role in it but even around my closest friends and family, I stutter all the time. Yet as soon as I'm alone, I can read the longest, vowel ridden words and sentences in existence without any issue. How I do deal with this type of stuttering? Is this just a confidence thing?