I have a feeling you may fear him, or maybe the trauma from the abuse triggers you to stutter when you talk to him. Could be a subconscious thing.