Same thing happens to me. Then you repeat the line, and people because don't realize it's a joke they're confused for a second until they do a mental switch to "oh, he's making a joke" and the entire situation which should've taken 5 seconds and ended with a laugh and increased connection now to ends with awkwardness and some disconnection (at least thars how I usually see it).