As someone who also stutters, it's pretty fair to not want to date someone who stutters. I understand why it's annoying. I don't mind it. At least they're not avoiding me because of my personality or character. Don't take it personally! If someone chooses not to date someone because of a physical/mental problem, it's hard to blame them, imo. It's a lot to take on as that person's partner. We just have to seek people who are especially compassionate. Good luck dude!