Of course its going to be hard, even taking care of young brothers is a huge task, while stuttering just makes it a plus for suffering. You should simply try accept the fact you need to give the best out of yourself. If you would to have no brothers you could easily accept the fate of no who. But having brothers that need a strong figure to respect. You no longer live by your free will, you determine their life and so it is your duty to make their life worth it just as yours. Try reading psychoanalytic studies which myself, helped enormously (for stuttering and self help)