Your worries are perfectly understandable. I think for his sake and your sake, just try to cherish that he'll still grow into someone will full of capability. People can have a stutter but they can still be happy, confident and push through on it. If it reaches an age where he can communicate his feelings to you and feels upset with his stutter then you can discuss speech therapy with him without sounding too alarmed, otherwise he will feel concerned as if he's "abnormal" but with the idea that it won't be a cure but only you doing this so his confidence roots up. Support is the greatest gift you can give to anybody.