Very late to the party, but I know this too well. When I speak in my native language (German) and English everything is perfectly fine. Especially American english with its very soft consonants is sometimes even easier for me to pull off than German. My problem is my third langauge, Japanese. I'm have a quite high proficiency and read stuff pretty effortlessly, but speaking it triggers my stutters so bad. It just has such a mechanical rythm to it with so many hard sound. I can literally read a book about Japanese politics, but having some small talk with the grandpa at the lobby is almost impossible.