Should I worry about proving people I'm not dumb/slow because of my stutter? Or just "fuck them"?

Should I worry about proving people I'm not dumb/slow because of my stutter? Or just "fuck them"? I think every stutterer in the world is worried about being taken as someone stupid because of their stutter. My question is: should we REALLY be worried? What are the consequences of not giving a fuck? What's the worst that can happen if I just say "fuck them, they can think whatever they want, idc". Will I be alone the rest of my life with this mindset? Cause no stutterer wants that.