I don't think it's even close to being at the severity level of a slur, but it is ignorant and unnecessary. "Did I stutter?" is another one that is unintentionally offensive and makes me cringe any time I hear someone say it.