I don't think it's a big deal; non-stutterers don't realize how rough stuttering is, so they don't take it too seriously. Honestly, that somewhat relieves me, because it implies that if others don't really care, I shouldn't either.