There are 2 factors here: people being rude due to your stutter, and your passion for physics. Those factors aren't related. Maybe consider reporting them to admin, or ABSOLUTE worst case scenario find a school like a commuter school where you may not have to socialize that much. But I'm not even in college yet so my word may not be the first advice to take.