I'm glad you find that comforting. I think it depends on the sort of look. Like sometimes they can look impatient or god forbid bored. Maybe confused, which I suppose is understandable. I'm not always offended by someone looking away as long as it's not obvious they're trying not to look at me or losing interest.