OP sounds young. Figuring out what it means to be a man is different for everyone. It comes to some earlier than others. If you have a low opinion of yourself, yes, people will look down on you. Your perception of yourself will influence how others see you. Maybe more than any other factor. When you stop feeling shame about your stutter, the ridicule stops as well.