I hate to see my stutter as the 'elephant in the room'

I hate to see my stutter as the 'elephant in the room' Several people actually revered to my stutter as the elephant in the room. I don't want it to be this big thing. Actually, letting people know I stutter makes things worse. I even stutter more. What do you guys think ?