i've had it happen to me multiple times. People think you are just straight up rude, but its also to long to explain in the short period of time. Best to just get on with your day. You probably won't see them again anyway.