The Romney campaign has actually emailed the full context of the speech out to its supporters. The problem with "you didn't build that" is that it sheds light on a ideology that's too focussed on collectivity and government. Bridges and infrastructure are helpful, but that doesn't negate the fact that the business owner is responsible for his or her own business. Arguably, the business owner actually paid for those bridges too, in the form of taxes.
Essentially, Obama's comment reflects his belief that government is the solution, not the problem. Republicans would disagree.
No, his comment reflects his belief that government is not the problem, nor is it the solution. Government is a tool that people use for their own benefit. In the case of business owners, they use Government programs to get loans to start the business, use the roads that Government built to move their products to market, and use the Teachers that Government hires to help them learn about our system. He was saying "you didn't build that
WITHOUT SOME HELP". That someone along the way gave you a hand.