People think I’m an immigrant/foreigner because of my stutter

People think I’m an immigrant/foreigner because of my stutter I’m an American with a stutter, and was born and raised in the US. However, I’m of South Asian ancestry. I’m usually a very quiet person due to my stutter, and only speak when I have to. Whenever I start to speak, it takes a couple seconds to start speaking, and so many people assume I don’t know English (I don’t have an accent, just a stutter). A couple days ago, a random person asked me for directions to somewhere, and when I started to say ‘I’m not sure’, they quickly said ‘oh no English, welcome to America’. Stuff like this has happened my whole life, does anyone else have similar experiences? Thanks