The more you focus your attention on it, the more it will impact your life. The reason you view it negatively is because you associated certain emotions to stuttering after many years. The way you improve your life when its something out of your control like stuttering is accepting it and changing your frame on how you view it.