I've had this too, and it made things even worse. Because some people who don't actually stutter might stutter when they are nervous, it led to this misconception that people who do actually stutter are untrustworthy.