I'm exactly the same with close friends and family. It's because I'm so unafraid of my stutter, I literally just say whatever I want as and when I want to. That means I say words and phrases I'd normally avoid or rephrase to stutter less, it means I'm not speaking with a guarded cadence or pronounciation. So I end up just naturally stuttering a lot more and getting really frustrated with myself.