If he knows that you stutter, you should have a conversation about how and why that comment is a trigger for you. If he doesn’t know, tell him.