I think to an extend Summers is right, in that men tend to gravitate toward certain disciplines, and women to others. There's not necessarily anything wrong with this, and it probably is innate to some extent. Women are less interested than men in math and science, while men are less interested than women in literature and humanities. There's nothing wrong with saying it.
As I understand it, Summers was saying that women are innately worse at math and science than men, which is quite different than what you said. What Summers said was simply false; what you said is simply a fact.
Nevertheless, I basically agree with the rest of your post, however.
I think that what I said is probably what Summers meant. I can't believe that he really meant to say that women were innately inferior in those disciplines. Sometimes, people say things and it just comes out wrong, and I agree that we ought to stop being so sensitive about it.
If he had said that men were inferior at language skills, feminist groups like NOW would applaud that, and showcase it as proof of female superiority. They would use it to argue that women should be favored over men. So I find their stance ridiculously hypocritical.