Its mostly psychological ,and based on your thoughts and habit You think in a way that makes you anxious while speaking outside ,while your brother is more outgoing ,I think then he doesnt feel the pressure to be fluent or didnt care about stuttering and judgement by other's. And fear and stress play a huge try to adopt the habits which are used to counter anxiety and fear and stress its will work definitely.