Control what you can control. You can't control how people react to your stutter. The good thing is that you're actually trying to talk. Keep talking and people will adapt to your stutter. When you're nervous, embarrassed, anxiety-filled, sad, mad or whatever negativity you're feeling, your mind defaults to negative thoughts. In reality, you just have to stay busy whether it's studying, learning, working out, or doing something fun. YOU want to be a Theoretical Physicist, so keep pursuing it and you'll happy you did. You keep doing your college work and everything else will fall in place. I know it's hard, but do the best YOU can. You stutter which makes stuff harder, but as long as YOU try YOUR best to speak you'll get better and things will turn out fine in the long run.