I would say that I experienced both cases. Like your case, at times I stutter less because I feel safer with company. Like your twin brother's case which is what I'm going through at the moment, I stutter less when I'm more nervous and really want to hide my stutter. I guess it's because I relax way too much and I genuinely don't care if I stutter or not when I'm relaxed(even though I do care while I'm actually stuttering) that I fail to actively reduce my stuttering causing myself to stutter more. When I have to interact with a person I don't want to stutter with, I tend to focus a lot on bringing my stutter down and I actually succeed.