Anyone else makes themselves sound dumb to make it through a coversation without stuttering?

Anyone else makes themselves sound dumb to make it through a coversation without stuttering? For example when I'm talking to people in English which is not my first language, I'd often sound like I speak broken English. It's such a shame honestly because it takes away the focus from the conversation and it feels like I'm just trying to make it through. It has gotten a lot worse through the years and I'm not sure what to do anymore.