I stutter more while talking to men.

I stutter more while talking to men. I have noticed that when I talk with female friends I don’t stutter at all, I tend to do more facial expressions and be more expressive in general, but when I talk with men (it doesn’t matter if I already know them or not), It is harder for me to talk. I’m male.