I think it just depends how much it bothers you.
I'm 23 and working, but I only have a couple grand in the bank. My family was and always has been middle class. Nevertheless, I don't look down on the rich or the poor. People obsess with money far too much.
He's rich. I'm not. Who cares? Just because someone made money doesn't mean that they "owe" somebody else money just because the other person might not have made as much.
What I'm asking has nothing to do with the fact that he's rich, only the fact that he keeps pointing this out, repeatedly, in a way that hurts his campaign. Romney's an intelligent man (part of the reason he's rich), and ought to know that people prefer a candidate that's humble to one that seems to be unable to go a day without pointing out just how much richer he is than everyone else. People generally don't dislike people for being wealthy, but they do dislike them for repeatedly pointing this fact out.
There are other politicians in Romney's category of wealthyness, who have gone virtually their entire career without the sort of "Look at me, I'm super-duper rich" comments that Romney makes on a near daily basis.