It's not just that they might misinterpret stuttering for being "slow" when you stutter it naturally causes a disconnect in the sacred Homo Sapien obsessed flow and dance of speech/communication. It immediately stirs up discomfort in people as we are wired to deeply analyze speech, as we are the only species to have language. To forge and bond relationships we subconsciously mimic another through speech and rhythm. Remember the words you say dont really matter, what gets analyzed in everything in between. Stuttering is associated with nervousness/anxiety which unfairly gets attributed to incompetence.