Do ignorant people tell you that "You lack self-esteem" ?

Do ignorant people tell you that "You lack self-esteem" ? ...because it happens to me all the time and I think it is due to my stuttering. I find it really irritating actually even though I know they genuinely want to help me when they say that.