Would I be wrong to move out in this situation?

The most important thing you can do for yourself is finish school. As long as you are not mistreated, suck it up and get an education. Go as far as you can in school. When you get to college, you can get housing and move out. Education is the only thing that gets you the opportunities you want. Take care of your brother and keep yourselves safe.