Conservatives? No, I disagree.
Racists, Nazis, and neo-Confederates? Yes, I agree.
What if they are doing this privately? You can't really criminalize someone from having hatred of someone else. Everyone hates somebody.
See, this is where your phony moderate veil falls apart. You pretend to be a true centrist and then someone says "society should not accept racists and nazis" and you respond with "well, wait a minute--"
You're not a centrist, you're an appeaser. You're fine with the status quo, even if that status quo includes making room for some of the worst ideologies our world has ever known. We should not accept bigotry. Not even privately. "Everyone hates somebody" is not a valid response to "racists and nazis have no place in society." If I were to say "Yankee fans have no place in society" and you said "well, Yankee fans would feel the same way about you, a Red Sox fan, because every sports fan hates some other sports fan" then you'd have a valid argument.
Just because you can't criminalize hatred does not mean that we should accept it as a society. There should be social consequences for bigots. I wouldn't advocate for killing or arresting nazis, but I am 100% okay with shutting them out so that they don't feel welcome, either in the discourse or in society at large. The fact that you're willing to split hairs and be a pedant about that is telling.
I'm a moderate. I believe in free speech. I also believe that people have the right to be idiots and if they are racists, that is their problem. They have the right to be idiots.
I think you need to be more clear whether you think these people should be welcomed/accepted by society or just if their viewpoints should remain legal.