Agree, it’s annoying. I tried to systemize all conditions which help me and make it more difficult to speak. There are too many of them to list. However, my analysis showed one noticeable factor (amongst others). If the person we talk is kind of (weak?) type our speech in s more fluent. And vice versa. Then I asked myself why. Besides pure psychological reasons, there is a plausible explanation. A person with benevolent attitude has a tendency to accommodate your speech tempo and rhythm. And this helps a lot, at least in my case