This happens to me also, for example youre talking to a friend on the bus and people start staring at you. I don't really know what to say other than, don't worry about it. If a person is enough of an asshole for that to be an issue, you don't want to get to know them anyway right?