Personaly, it doesn't bother me. It gives me an opportunity to explain my stutter to people who are uninformed. Humans are curious beings and some just don't know how to ask without coming off as rude.