The northerners were much more racists than the southerners. The southerners considered the slaves fully human, while the notherners only considered them 1/5 human (the 3/5ths decleration was made to please both the southerners and northerners). The southerners actually used to baptize their slaves, and brutality was very uncommon. The northerners did not consider the slaves as equals at all, and didn't want to have anything to do with them.
The northerners really wanted to keep the southerners an agricultural society- and they succeeded. They put the southerners in a very awkward position, forcing them into independence. This gave the northerners an excuse to attack the south to keep the south an agricultural society so that they wouldn't compete with the north.
As a matter of fact, the Vatican sided with the south because the northerners were so power-hungry and racist.
So no, the confederate flag is not a symbol of racism- it's a symbol of how the south was abused by the north, and prevented the south from being what it truly was.
Wow. Talk about reactionary southern propaganda.