I have the same with family members. Maybe your self-image as a stutterer is stronger when you're around them, because since your childhood, you've spent the most time with them.