I am the exact same way, and think (like someone else said) it's that im not comfortable with my own stutter so I get frustrated at the other person. Though its very rare to talk to someone else with one.