We all have different behavior sets for different contexts - and stutter tendencies often change, depending on set/context. It might just be that you have a certain distance from your usual behaviors, when talking with peers of the other gender.