In your position I'd explain that I appreciate the effort that he's putting into talking to you. Even as someone who's accepted and can deal with stuttering, conversations still take effort, and it's a lot more stressful without that perspective.