It is a much bigger deal for yourself than it is to others. If the only thing holding you back is your stutter, then you are able to do a lot more than you think. At times it took me a minute (which is insanely long) to just say my name and it feels awful, but people forget about it almost instantly. Really the thing holding you back is you blaming your stutter for everything bad happening.