The annoying part is that they don't look at you when you stutter? For me, this can be very disappointing and it can give me the same thoughts you have. But also, for me it's easier to talk with people if I don't look at them - maybe you should try this, might this way you won't worry about what they're doing/thinking as you don't see them. Plus, I think they're not necessarily want to be rude or think that you're not worthy of talking with. Probably they don't know what to do in this situation - they might think that it'll annoy you if they stare at you, or just simply really don't know how to react, and this way, they can "distract" themselves from the situation.