So, what am I missing? Are we seeing huge effects of social structure (larger households which allow infection even under lockdown)? Is there some impressive contact tracing going on in the USA that I am unaware off? Are lockdowns simply stricter on that side of the pond (and what about Sweden then)?
This is the most likely answer. Much of continental Europe is densely populated and have multi-generational households that increase contact between at-risk groups and the general population. Furthermore, countries like Italy, Spain, and France have top-heavy population pyramids. They also have less trust in government (esp. Italy), which is why you hear so many stories of people not taking lockdown seriously.
Countries like Germany and Sweden also skew older, but have less multi-generational households and have much stronger trust in government and people are more likely to follow social distancing directives.
Meanwhile, the U.S. skews much younger than virtually every European country (IIRC?), has fewer multi-generational households, and the American lifestyle (lack of public transit, high use of personal automobiles, more single-family homes) also inherently has a degree of social distancing built-in. This is despite a relative lack of trust in government like southern European countries, which helps explain why deaths per million have been lower. Finally, the U.S. was hit much later than most European countries, giving Americans some time to prepare and think about how to change their lives and learn what social distancing means.
And of course, statistics are not perfectly comparable due to the different methods used.
Every country is completely different.