Honestly people do care. Most people are just nice enough to not mention. As adults at least, most of us know how kids are. I'm sure that they do talk about it. Not obsessively, but in the same way that people talk about other people with defects like lisps or something. It's rare enough that people often have no idea how to react too. And let's be honest, society values speakers that project confidence. A lot of people mistake stuttering for nervousness or timidity.