It is possible that he stutters more as he gets comfortable with you. Don't think you're making it worse. Some stutterers stutter more when around family and close friends because he/she feels safer to stutter.