I think it mostly has to do with mirroring. We unconsciously imitate the person we are listening to so we can emphatize and build rapport while also expecting to hear a certain language, pitch, tune etc. When the other person starts stuttering all of this breaks down because now, instead of having a pleasant easy-to-follow conversation, you get this mess in your head that you have to deal with, try to figure out what's wrong, what the person is trying to say, how to respond etc. So the most common reaction becomes a wtf face, looking away or just staring blindly. Some people even start stuttering themselves. This is not due some kind of character flaw on their part, ironically it's the opposite. They feel your pain and it is making them uncomfortable.