There's a cool mindset called "let them, let me" that you can look into but I also experienced the same with my stutter where the less I care the easier it is and the more I cared like with new situations, the more difficult it may be.  First I'd start caring less after the fact, after the stutter, then it moved towards the middle of it and then I wouldn't think about it anymore.  What ever happens happens, "let them, let me."