If people like or know you for who you are then they don't find your stuttering annoying or frustrating they'll see you as a person beyond your speech impediment and also if people don't know about you that much they'll definitely find it annoying. It totally depends on a person's patience level.