Nathan, you are supposed to use a pin punch to tap out the roll pin. It is just slightly smaller diameter than the hole, very hard, and has a flat end. A pointed end (such as a nail has) is bad because it tends to expand the roll pin and make it harder to get out. Even with the right tool they are a bit of a pain to remove and install.
