You can't compare same-sex marriage to polygamy.
There is a difference between prohibiting a certain class of people from entering into a legal contract (which is what banning SSM does) and limiting the number of such contracts one can enter into (which is what forbidding polygamy is about).
I don't see that logic playing in the average Joe's head. Marriage in most people's eyes has to do with love, not a contract. Hence their support of SSM/divorce. (Simplistically: Tow boys should be allowed to marry if they love each other/if you aren't in love, you shouldn't have to stay together)
Once they get the idea you can love two people at the same time, they'll think it's ok.