On the surface, the difference between 2000 and 2004 can almost entirely be explained with Bush's strong performance among women voters. In 2000, Gore beat Bush 54-44 among women but lost men 43-54. In 2004, Kerry largely matched Gore's number among men, losing them 44-55, but only managed to win women 51-48.
Bush also did remarkably well with the growing Hispanic vote in 2004, only losing them 44-53.
This, the "security moms". Kerry did about as well as he could have. Voters were not going to vote out Bush in the first post-9/11 election.