I always heard the same thing, and I use to stutter alone, and it was very defeating hearing others did not. I feel there are two parts to stuttering, the internal and the external. I always felt when I stuttered alone it was more external, I was in the habit of speaking a particular way, that produced a physical tension and a stutter, even without the added stress. However some physical stress was clearly there and of course the thought and habit and stuttering was there. I slowly changed the physical way I spoke and even more slowly unraveled the internal until I eventually became fluent and free in my mid-40s.