I'm going to go ahead and assume your boyfriend is not a horrible person per se, but that he doesn't quite understand how not okay that was. Have you explicitly talked to him about this before? It reflects very poorly on him to say something so selfish as "having to deal with" YOUR stutter.