My stammer is the same and I only speak about it to one or two people. I would strongly urge you to tell him, he probably won’t even care that much. I’d also suggest trying to change your mindset and being more open about it, it’ll be better for you in the long run. Unfortunately I can’t do this myself though but I am 38 so likely a bit older than you.