You've spent the most time with your family members since you were a child, so you have a strong stutterer-identity around them (and you're afraid what they'd think if you suddenly don't stutter around them).