Saying that only men can rape is more mainstream because they define right and wrong around "pOWUr diFerEntIAlS" instead of the merits of what a person does
Even by this definition that assertion fails, because a rapist clearly holds inherent power over their victim by virtue of doing that to them.
I agree, but that's the problem with all these "power differential" based moral arguments. If someone does a thing then obviously you had the power to do the thing. It's more self evidently wrong to make that argument about rape than rioting or hate speech to name a couple examples but the argument doesn't make sense in either case.
The "power differential" thing makes some sense if you're analyzing the cumulative amount of harm that one person or group can do to another by performing the same action or set of actions, but increasingly I think that that's too consequentialist for me to be comfortable fully agreeing with. And I've certainly never had any truck or trade with applying it to sexual abuse of any form.