It can happen, personally, I've experienced the same. Maybe your self-identity as a stutterer is stronger around people you've spent the most time (and stuttered) with.