This is probably not the best advice but I've always found that the best way to deal with bullies is to start swinging. Even if you lose, they'll think twice before fucking with you again. And if they do decide to do it again, swing harder. I would even encourage you and your brother to take some self defense classes. Stutter or no stutter, nobody is gonna fuck with someone who can put them on the ground and break their arm.