What I have realized is that this is probably their first time encountering someone like us, that stutters and they don''t know how to react to it. I usually say something like "btw I your ears are not faulty, I have a stutter" and then I let them make the next move. Some people will disengage and that's fine; others will stick around. Those are the ones we can communicate and hopefully build something with.