I feel the same way , i have always been self conscious about my stutter but the people around me never really notice it or it doesn’t bother them. I too think maybe it is pretty privilege maybe you get away with it because people don’t mind it at all coming from someone they are attracted to