I don't see how proposing abolishing the fed is crazy. We functioned fine before we implemented quasi-central banking, and a lot of our problems now can be traced to the irresponsibly low interest rates Greenspan implemented. I'm not sure if I'd back having totally private currency, but obviously we have to overhaul the way we handle currency, loans, etc. Other than that I agree, Paul gets too much of a pass for his association with the Birchers and neo-Nazis like Don Black.
I'm not saying Paul is a total fruitcake, in fact I'm prone to agree that abolishing the fed is a good idea. However, I don't consider him "libertarian", everytime I refer to Dr. Paul I call him either a traditional conservative or a paleoconservative.