In truth, if O'Malley were actually to take Clinton down and win the nomination, he would have to do it in a way which would damage the party's chances for the General Election. He'd have to trash her personally with stuff that would stick. He'd have to go after her on Benghazi, and give Republicans credibility.
If he did this, he'd be done in politics. He'd be the guy who blew it for the Democrats, the guy who ruined their brand just when it became rehabilitated from the McGovern debacle of 1972. He'd gut the blame for a Republican President appointing new Supreme Court Justices who overturn Roe v. Wade and who uphold Citizens United. He'll get the blame for the repeal of Obamacare, for additional Lindsey Graham wars, for the trashing of the safety net, because if he trashes Hillary enough to win, he'll have to do it to the point where it will be a GOP landslide, with the Democrats winning only the Northeast, the Pacific Coast, and IL. In reality, O'Malley would have to wreck the Democratic Party to be nominated. George McGovern did it in 1972, and look what happened to the Democrats ever since.
The Clintons did the same thing with Obama dug up Reverend Wright, the birther stuff, Muslim rumors etc. the Clintons don't hesitate to run nasty personal campaigns O'Malley shouldn't hold back.