Perhaps you need to redefine what you mean by “ruin the whole thing”. When I think of what it means to “ruin” something, I think of destruction, and this only happens in a conversation when something hurtful, insulting, deceiving or malicious is said. Stuttering doesn’t ruin anything unless the person on the other end of that conversation is dumb and shallow enough to let it affect their perception of you (and regardless of how fearful you might be of this, the vast majority of people truly don’t care). Nobody is more critical of yourself than you, so give yourself a break! Think life-giving thoughts and realize that self criticism does you no good in these social situations.