Would it be rude to say I like someone’s stutter?

Would it be rude to say I like someone’s stutter? We work in the same industry and talk on the phone a lot but he always apologizes for it. I find it to be really soothing. But I don’t want to be inconsiderate. Would it be better to say don’t apologize or is that worse?