I can relate with the sudden/direct questions and mentally frazzled bit. I just automatically say umm to help me reorientate it a little and buy some time. Idk why it can be so destabilising at times. For me, I usually am already thinking of what to say in the least “stuttery” way when having a casual conversation so I guess when thrust into one by a sudden/direct question, which may have its own time constraints, that can rob you of the time needed and add extra pressure. In the end, it is super frustrating at times but it is what it is.