Correlation isn't always Causation. It just can happen that you are bad at math AND have a stutter, I always meet people who cannot comprehend basic logic and math but have a fluent speech. Even the actor, Rowan Atkinson, who played Mr.Bean is said to have a stutter,but he also has a PhD. And a degree in electrical engineering which involves good amount of math and logic,so that's also something to think about.