Well if you care that much about his thought or depending on how offended you were make it a deal. Let them know that you are having a conversation and that turning away is rude, from there you can decide which way it could go. Seriously though if your talking to someone and you stutter and they start turning away like they are not wanting to talk to you then that’s fucked. Plain and simple.