I'll try to provide an explanation for you. It has been shown that at rest, people who stutter have less blood flow to the left inferior frontal gyrus. This is the region responsible for speech production. Keep in mind that speaking is a metabolically expensive task for the brain. So when we are by ourselves, there isn't much load on the brain, but in situations that are more cognitively demanding, the brain is unable to meet the metabolic demands that the neurons require. This is pretty exclusive to the speech regions, other regions are intact. So we may look at anxiety etc as the enemy, but really its the inability to meet the metabolic demand that the brain requires during cognitively demanding tasks because we're already at a deficit at rest. There's absolutely more involved but this is a short explanation. If you've ever exercised and had immediate fluency during or immediately after, its because exercise increases blood flow to the brain.