I guess that's is because the moment we hear other person stutter we are focused to hear how we sound, and it is awful. Usually when i record myself in a conversation I think: "that's how i sound?, I thought i spoke better than that" xD.