I've always thought it's because you lose all the nonverbal cues that come along with in-person conversations. Like /u/middletwin said, if you have a big pause, the person on the other end can't see why that is. Your effectiveness as a speaker seems a lot more important on the phone than your effectiveness as an overall *communicator* because we only have the speech to rely on in that medium.