I still advise you to say it, because unless you can hide your stuttering perfectly (which is not always possible, or sometimes exhausting), if your listener does not suspect that you have a stutter, that could create an even more negative misunderstanding (I imagine I'm not the only one who sometimes fears being seen as stressed, slow or incompetent when people don't know I stutter)