Wouldn't you also say the emense gov't support required to prop up slavery was a form of protectionism? Not to mention also the violation of freedom caused by the fact that poor white farmers were practically conscripted to catch runaways and/or would be called upon as part of the militia in the event of a slave revolt. A slave society is by definition a police state by necessity of the fear of slave revolts. It was a crime to advocate against slavery in many Southern states as well as you get closer to the Civil War on that very basis as well.
I agree. The slave system had similar effects to protectionism, too, with the Southern economy becoming highly undiversified and reliant on the export of a small number of agricultural commodities (which is why the average Southerner would've been hurt so badly by the Republican tariff hikes). In addition, the wealthy slaveholders were similar to the rent-seeking businessmen in the North who sought higher tariffs to protect their industries. Both groups benefitted from the profits their distortionary systems provided, but the North and South as a whole were both poorer as a result of their respective activities. As you noted before, slavery was really not beneficial to the average Southerner, which is another reason why it isn't fair to imply that ordinary Southerners "got what was coming to them."