I think this is cause your larynx isn't involved when whispering. Hold your hand to your throat when you talk normally and you'll feel it vibrate. Now try whispering, and you'll see it won't.