Are you comparing the Confederate flag to the Nazi flag? Seems that you are since you are lumping the two together. More crimes have been committed under the Stars and Stripes then the Stars and Bars.
It's all about perceived meaning. To a Black person in this country, the Stars and Bars is a slap in the face. To a person of Hindu faith, the Swastika is a revered religious symbol. Our Indian neighbors across the hall had swastikas drawn on their doorstep. It came as a shock to us (as we have an obviously Jewish name, and thought it might be against us), until we realized that they had no intention of offending anyone. Likewise, to YOU the Stars and Bars may be a symbol of rebellion against tyranny, or the cultural heritage of the South. But to a Black person, it is a symbol of slavery and racism. That is why I lump the two together.
I will grant you, the intent of those who fly the Stars and Bars usually isn't to advocate white supremacy, whereas that's almost the universal intent of the Nazi flag. But since I believe both are protected speech, does it matter?