You can do nothing. In my job I also noticed that the better my speech is the "more popular" I am. When my speech is super bad, nobody talks to me much. They mean it good, they just dont want to make me unconfortable. But they always treat me nicely and friendly, regardless of my current speech. Just be nice and friendly. Some of them will like you for your friendliness and some will not care much, that´s normal.