Unfortunately, I think it really depends on how attractive the stutterer is otherwise. If you're good looking and/or confidant, it's probably not much of a problem at all. If these aren't the case, it's probably a compounding factor.