There are lots of reasons why someone could be afraid of public speaking - fear of looking stupid, fear of being judged for what you say or how you look, fear of forgetting your talk, fear of saying the wrong thing, etc. Fluency is one fear that's unique to stuttering, but can you imagine a world in which an otherwise fluent person would have anxiety for another reason?