I think people rarely feel annoyed, just that it is akward. The key for me at work, always, is to break the ice. Joke about your stutter or be open about it, I mean Its not something you can hide anyway.