I don’t think so. There are comedians that stutter. Yes, part of humor is getting to the point before the other people can guess what your point is. But as long as you commit to the joke, I think you should be fine. And tell them that you like to tell jokes sometimes and don’t want people focused on your stuttering. The problem I think is that because you feel bad about stuttering, the listener goes into empathetic mode rather than laughter mode, and that is maybe why you might not get the best reaction. Committing to your jokes is part of letting them know that you are self-sufficient and they don’t need to feel bad for you. The other part of it is telling them that you hope your words aren’t taken differently because of your stuttering. Then they can go more into laughter mode.