Mikulski had a respectable career. I could understand why she might not want to run again in her 80s, especially with the prospect of being in the minority.
O'Malley doesn't have much of a shot of the presidential nomination, so it would make sense for him to run for this office. It would mean his political career doesn't end with his Lieutenant Governor losing to a Republican, and it makes him more appealing as a candidate in later Hillary-less cycles, with executive experience and knowledge of Washington. He'd have a platform as a sitting Senator rather than as a former Governor.
I think O'Malley sees himself more as a President than a Senator. I think he will run for president, and challenge Hillary Clinton anyway, but he will be asked to run for Senate, but he won't do it for his presidential ambitions.