I have the exact same experience. I suspect it's because I'm on 100 around strangers, constantly micromanaging my breathing and posture and using every trick I know to appear as fluent as possible. But that's exhausting to maintain, so around people I know I just speak as I would casually, stutter and all.