I think most people are just surprised when they meet a stutterer because it's so rare. They don't know how to react. And, this sucks, but stuttering can be mistaken for nervousness, so they may just assume that the person is anxious.