You're right it does interrupt the flow of conversation, but a positive attitude and confidence go such a long way. If you can't control it, do your best not to let it bother you. Nearly everyone (adults, that is) will listen more closely, fluent or not, to a confident speaker. I used to think it was impossible to stutter with confidence but speech therapy and practice worked tremendously for me.