Doesn't the grit wear away the faces of the tool where it grips? That's my main reason I clean them after each use. There is supposed to be a thin layer of hard steel before it becomes prone to wear.
Just makes sense to cleanup you tools instead having to reach into a tool box full dirty grease.

Unless you like reaching to used oil pans noodling (hand fishing) for tools.