I'm sure if state recognized it'd be used far more frequently by people trying to abuse the law than as actual "marriages". Yeah I know some people screamed the same thing about gay marriage but that didn't allow for any type of marriage of convenience that wasn't already possible between a man and a woman.
Agreed,
The primary difference is that a gay marriage is still a lifetime binding unit of two people, albeit of the same gender. So the same legal framework used for centuries with respect to marriage could still apply easily. One salient one is the permanence of marriage - while divorce is always an option, it does bind one partner to another in a legal fashion, just like a heterosexual marriage.
One of the problems of polygamy, that you're correct to point out, is that since one person can be "married" to multiple others, the basic pillars of marriage - mutual commitment, fidelity, etc. - have the potential to be eroded strongly. I could still see a case in favor of it, but it certainly has far more ramifications than the SSM debate.