Maybe that's around your circle of people. In my case it never happened like that (started stuttering at 7, I'm now 25), maybe you should try to explain what stuttering really is, maybe they will understand that you're not different. Honestly it all comes from the fact that they are ignorant of this disorder.