Not everything is down to having a stutter. They might just not get along with you. Just being outgoing doesn't mean you'll always get along with everyone.