There could be multiple reasons for that. This could because you value their opinions of you very highly so that puts pressure on your to speak "perfectly". Which will add tension and cause you to stutter more. Or certain people can trigger certain traumas your holding onto. Or it could also be that these people view you a certain way "shy, hidden, repressed, quiet" and when you're around these people, that side is shown, and that side of you will always stutter more. Any of these makes sense to your situation?