Ehhh…I have mixed feelings on this. First, I agree with you. It’s a tad insulting and weird if you think about it. But, I know from experience that the person saying it truly means well and is trying to let you know that they aren’t bothered by your speech impediment. I know it’s strange, but from his perspective it’s meant to compliment you. Maybe let him know how you feel to give him the opportunity to defend himself, especially if you think there’s a possibility for a connection.