On the contrary. Not everyone feels this way. As long as it’s acknowledged in a respectful way, this could lead to a better connection. Some people feel at ease knowing they no longer have to try not to stutter— the other person already knows, no need to hide or feel ashamed anymore. By the looks of it, OP is very considerate and should have no problem discussing it, giving the person who stutters an opportunity to share what he is comfortable with. Of course, if this person asks you not to talk about it, respect their boundaries. Personally, I don’t mind if people ask me about my stutter. It makes my tension go down, making the conversation easier. It gives me the opportunity to share insights about stuttering most people may not know, helping me feel understood.