This is common, if not universal for stutterers. The mechanism that causes stuttering (in the physical sense) appears to be somewhere in the voice box / vocal cords. When it isn’t activated, such as in whispering, there’s no disruption. If there was, I suspect there would be fluency disruptions in breathing. FWIW, the speech therapy program I went through used whispering (“talking like a ghost” is they called it) as a first step. After that they taught us to speak in a soft monotone (“robot voice”), followed by slowly adding inflection (“talk like Marilyn Monroe”) to our speech and working up by adding more inflection, natural speed variations and less breathiness as we were able to. For me it worked. I’m mostly fluent now, although like all of us I still stutter from time to time. I’ve just learned to roll with it and get myself back in the right path.