It's the nature of the beast that you don't struggle when talking to yourself in an empty room. I'm a very severe stutterer on bad days, but totally fluent when talking to myself. My advice would be, though it might sound paradoxal, to accept that you won't be fluent 100% of the time. That might stop your struggle which results in more fluency.