Yes you will have to remove it all, not a nice job and getting the wheels off that taper is not an easy task. At least you have the wheel with the sprocket off that can be the hard one. If you jig it up properly you can weld it and get it pretty straight, but it needs good preparation for the weld to be strong enough again. It can be done but it is not just a case of put it back together and weld it up.
