Most likely because you actively *try* not to stutter while with others. Try speaking before the thought of stuttering kicks in, your speech will be fluent.