It's not just the speaking that's exhausting for me. It's the whole of social interaction. I think it's why many of us become introverts. Just being out and talking to others is draining and we want to retreat to a safe space. But while that may seem like a good solution in the short term, it's not in the long term. Withdrawing yourself from any social interaction makes you more isolated and will ultimately be much more draining long term. I would suggest to try and stick it out. People don't care about the stutter as much as you do. Try to be friendly and make an effort to maintain your social ties.