It doesn't bother me when I hear this saying, but plenty of other things do irritate me. I think the most irritating for me, is when I'm in the middle of a sentence and the other person (always a guy, I'm a girl) asks "Am I scaring you? Am I intimidating you?" It's like... dude, you wish you were really that 'scary' to make me stutter like this, don't stroke your ego. I JUST have a speech issue, you're not that cool.