I imagine that I stutter more than you do based off your description, but I only mention my stutter to people I know I'll be having more than a short conversation with. When I do tell someone, I don't make a big pronouncement of it, I just say "oh btw..." They'll probably forget about it until I actually start stuttering but then they'll know to calm down and that I'm not having a seizure lol. But again, if I'm only saying a word or two to someone, there's no need to mention it. And if I do stutter with those people, they usually catch on pretty quickly and are patient, I've found.