Hi AVB,
Boring it out is not going to weaken it, you are only taking 1/16th of an inch out of the wall thickness and you don't need to broach the keyway out, a grinder will quickly take 1/16th of an inch off a bit of keysteel