As someone who now has a mild stutter but was quite severe all the way into my late 20s, my worst fear is that I will tell a fellow stutterer that I too am a stutterer, and they won't believe me. You truly never know what some people have experienced. That said, if you think these people genuinely have never had a stuttering problem, I would let it go. They will eventually realize after hearing more of your stuttering what you really meant, and they will be mortified. I've found throughout my life that some people genuinely have no idea how to respond appropriately to a stutter. I would give these people the benefit of the doubt and assume they're trying to make you feel better.