Same with almost anything else, some people will care and some won't. There is no universal answer. I've found that one's attitude about stuttering can be as important as the stutter. If you are confident and treat it like not a big deal others will sometimes follow your lead.