If you are not doing anything to fix it, then you likely are not thinking about it as much as the person consumed by thoughts and actions related to their stutter. Each approach has its pros and cons. I think the key thing you mention is "over the years." I have found that, as I age, my stutter has lessened. I attribute it to not judging myself as harshly, caring less about what other people think, and an overall increase in positive self image - all of which seem to naturally come with age.