I've talked to women on the phone while stuttering and none of them really minded. One of them actually said it helped her feel less insecure about her own speech issues.