I'm first going to mention the obligatory cliche "if it bothers him, you don't want to be with him anyway". If you're very anxious about it, you may feel better telling him. You don't have to make a whole thing about it if you dont want to. You can casually incorporate it into conversation. For instance , if you're talking about winter, which sometimes leads into everybody getting sick, you can say something like, "yeah I hate getting sick, it ends up making my stutter worse". Maybe come up with a more positive example, but you get the idea.