I think you are aiming a lot for perfection which subconsciously puts you under a lot of pressure to feel forced that you need to be smooth and consistent in verbal communication. People can have flaws in their speech as well of lots of "Uhhh" and "Ummm" that we tend to actively tune out and not mention it to them but this is not to say you aren't allowed to voice your struggles. Not necessarily a bad thing in being a perfectionist but it does sound like although people treat you with respect and decency regardless of small bumps here and there in your speech, you seem to be very critical and harsh on yourself that I personally think that crosses the line. I think it's something worth discussing by arranging a speech therapist to not see them as a cure for 100% speech fluency but to explore your emotions on this then try some of the techniques they suggest at your own will if you feel very uncomfortable at the sound of your stutter but again, not to see this as a cure.