I have talked with some fellow stutterers and we came up with a few ideas. One thing might be that when talking in person and you have a block (or what have you) the other person can see what is happening. On the phone you don't have that, you just have dead air so the other person might not know what is happening. The other possibility might be that we can't see the other person's reaction and that is a point of extra stress. We don't know if they are waiting patiently or are quietly laughing at us. While they are most likely waiting patiently, we can't see that so we might get a little panicy.