This is just guessing now but since it only happens with people close to you, could be because you care so much for them? So much even that you want to be "perfect" for them? Do you feel ashamed because of your stutter and do you try to hide or suppress it?