Yes, this has happened to me a lot! I’m from Puerto Rico and when I still lived on the island it happened fairly frequently (I worked customer service, so I spoke to a lot of people all day). And now that I live in the US, it still happens sometimes when I speak Spanish with other native speakers. I’ve asked my friends if I sound like Spanish isn’t my first language, and they’ve all agreed I sound very clearly Puerto Rican. I’ve ever recorded myself and can’t hear an accent. So I think some people have never met a stutterer before so their brains just jump to it being an accent.