I was put off by how long the school/training was (4 years med school + 4 or more of residency) and the general unpleasantness of dealing with hospital administrators and insurance companies and all that. My dad is a physician and I think he's a case study of what happens when people go into medicine more to please their parents than out of any real love for the profession.
Those are my biggest put offs as well. I'd hate to be in school for that long and plus dealing with the insurance companies would suck. Ugh. The on-call schedule, the long hours (this depends on your speciality though), the very significant student loans (hundreds of thousands of dollars), and the stress of someone's life being on your hands are also downsides.
Some upsides though include the great pay. Doctors can make anywhere between $150,000 to $700,000 a year (depends on speciality). The respect doctors recieve and the fact that you're saving lives (depends on speciality) are also upsides. Medicine is also great if you have an interest in anatomy (which I somewhat do, but it's not my main interest).