If you have a severe stutter and can almost never get your words out, then yes, it will be a turn-off for the other person. If you have a mild stutter or if you stutter from time to time, you honestly should have nothing to worry about.