Keep playing the sport if you genuinely like it. But distance yourself from these people because they don't seem that high quality of friends. Also fuck this clique. Cliques are for people in high school who desperately cling to a certain identity and are too afraid to follow their own lead. Following the lead of a group that doesn't want you and mistreats you is good way to fuck yourself up. Don't allow yourself to rationalize why someone rejects you. Get this stutter, height stuff out of your brain. Overly thinking about this stuff and using it to blame tough circumstances isn't healthy at all. A good way to stop overthinking is to actually become dumber. For example if Forrest Gump got rejected by a group or a girl he probably wouldn't think much of it and would go about his day. Forrest would probably leave this group and find people who connected with him better.