Knowing that you will stutter increases anxiety, so you consciously or subconsciously, refuse to speak a lot to avoid that. If you think that you need help with that situation, I think therapy can be a good place to start. Most likely, the plan will be to make a list of the least feared situations to the most feared situations, and go tackle them one by one, and discuss that with your therapist.