In my opinion, Totally ignore it, and don't change your behaviour, because whatever you're doing naturally is making him feel relaxed and comfortable to speak fluently, keep being yourself. I've been in a similar situation where my stutter got worse once I was aware she's aware of my stutter.