That can be a good way to think of it. I remember thinking, in highschool after the incidents where they stopped the teasing, anytime anyone from that social circle would comment on my art (I drew in class, everyday), or asking me about a book I was reading.. any sort of nice comment was met with me being pleasant but thinking "What do you want? Why are you talking to me?". I never trusted those folks again. I am glad you have friends that you can trust. A stutter doesn't matter to the majority of people. Sure, folks who may not encounter stuttering in their daily life may react in a less desirable way. Usually not out of malice and they stop their behavior. When I met stutterers for the first time at a NSA meeting, I found myself thinking the same impatient thoughts I had heard all my life "Talk slower. Spit it out. Sound it out..." and that shocked me. I had never heard anyone stutter worse than I did.. but that exposure changed how I think about my own stutter and humbled me. Meeting people who react badly to your stutter may be having a similar experience to me.. and once they have processed it, they get over it. Of course, if that continues, then by all means get that negativity out of your life.