Don't listen to them. I have stuttered as long as I can remember. People/society gave me the implicit expectation that with enough time and effort I would no longer stutter. I have improved immensely, but I definitely still stutter after 25+ years speaking. Having the expectation that it will go away one day has been one of the most detrimental things to my experience with disfluency. Don't give up, but don't expect the problem to ever go away. You will set yourself up for disappointment.