I would take them at their word. Probably 80% of the people I work with don't realize that I stutter (until I do). But I find it particularly frustrating when people think that I'm stuttering because I'm nervous.