I’m not sure if this will completely answer your question, but it’s possible you have some fear and perfectionism when it comes to speaking English. You are aware that your English skills aren’t as good as the people around you, and it seems like you may have fear that they won’t understand your accent. My suggestion would be claim a mindset of patience for yourself, where you are willing to calmly say something three or more times until they understand the word you are trying to use. And also don’t want to fully assume, but maybe also claim a mindset that you are as important an unique as everyone around you. This patience and positive self belief will help you stutter less, then you will be more likely to join events and hang out with English speakers. And as a result, improve your English by asking them questions and hearing how they speak. I can also relate. My native language is English but I speak some Japanese. I stutter in Japanese more than English because the fear and anxiety of having limited vocabulary, taking longer to formulate sentences, and just overall having the negative self belief that I’m an outsider in the culture, and am not following cultural rules correctly. I hope also that I will gain more comfort and take it more easy on myself while speaking Japanese. Not sure if this answers your question but thought I would give some advice based off my own experience speaking a second language