I think a lot of it is in how the person means what they say. If your colleague was saying it in what you perceived to be a nice way, then fair enough :) For me, the really intensely patronising way someone sad they find my speech issues adorable and cute really hacked me off. And yeah, I can totally pick up on when how I talk is annoying people and when they are becoming impatient with me, which makes it worse because I get flustered, ashamed and self-conscious.