Breckenridge. Secessionist bastard.
EDIT: I should probably include Aaron Burr, the bastard who killed Alexander Hamilton. Who else do I hate? Maybe William Rufus King. But really, Burr is ridiculous.
William R. King? But what on earth did he do in first place?
I guess it's just that he was super pro-slavery, not that that's an outstanding characteristic of many politicians from that time I guess. 
Cut him some slack, he died so early in his term that he didn't have much time to do any damage.
My answers would be John Calhoun (also super pro-slavery), John Breckinridge (for the reason Cathcon highlighted), Andrew and Lyndon Johnson both (although they were both even worse presidents and people), and Henry Wallace (Communist sympathizer). Maybe Hamlin, too.
