As a guy, i can tell you with 100% certainty that this sort of comment comes from sexism. There's a lot i could talk about it - how a lot of it is structural, and men don't even notice they are being sexist - but the important part is that you can rest assured you are on the right in this. There is no fcking excuse on the men's part. Even when it comes to dating, that is just a small part of a way bigger issue of how we feel when talking to other people.