Didn't Costello have the numbers just after the election when Howard fell on his sword?
The numbers, yes, but the time wasn't right. What happened to Nelson would have happened to Costello if he'd taken Leadership (or Turnbull if he'd won that leadership ballot). It's what happened to Latham as well.
Costello knew that if he took on the mantle of leadership, he'd have been a failed Opposition Leader and would end up retiring from politics - which he didn't want to do. Instead, he retreated to the backbench to wait. If he didn't have any intention of ever taking leader, he wouldn't have turned down the plethora of job offers he received.
Not challenging for leadership against Howard was because he didn't have the numbers. Not challenging for leadership post-Howard was because the time's not right.