It's sadly to be expected. Politics will attack any perceived weakness you have. Weight, looks, any misspeaking at all, etc. Sometimes I feel it's mostly about that and less about the actual politics.