It's not your fault. If your stutter bothers them so much they don't want to be your friend, that's a problem with them.