Take care of yourself. If you're in a point where you feel the right way to proceed is to not give an f about your stutter, proceed as you see fit. It's a problem that you're making other people's opinion a priority in your decision making. Further, nothing is binary in this world. Everything's on the spectrum. It's not just two options.