Because it's not an anxiety of speaking to someone you're afraid of, but it's a speech anxiety. You can experience it while speaking with literally anyone.