OP, I’m sorry but this is not something anybody can tell you. You have to decide which concern takes priority, because ultimately you’re the only one who knows. If you do decide you can’t stay with him, my advice would be to break up ASAP so he can move on with the rest of his life.