I reckon that school gives stutterers a skewed view of how we are perceived. Since graduating, I've had very few issues with people and my stutter improved to where it's sometimes barely noticeable. The OP just needs to tell the person. If it makes them stutter more, then so be it. Chances are that you are right; if he likes them, it will almost certainly not be a deal breaker.