The commerce clause does not authorize the regulation of "anything remotely related to commerce" (which basically means "anything at all," because everything can be said to "have an impact" on interstate commerce). It only authorizes the regulation of commerce itself: "The Congress shall have Power ... To regulate Commerce with foreign Nations, and among the several States, and with the Indian Tribes."
Marriage may be remotely related to commerce, but it is not commerce.
By current judicial interpretations of the Commerce Clause (say what you like about it, it's still established law), the Commerce Clause does indeed apply in this case. PA's and DE's recognition of a same sex marriage granted in NJ effects pension distibutions, health insurance, estate planning, and all sorts of other commercial activity, across state lines. The Commerce Clause has been invoked to justify federal legislation on much flimsier grounds than this, and has been so for at least 70 years.
Has Commerce (I can never remember whether it's spelled with a c or an s, but maybe that's because I read too much UKian lit) been over-extended? I certainly think so. But under currently settled law, Congress does have the right to regulate this.