I think only the true racists care about this kind of stuff.
Part of why the coverage on this issue is so out of whack is that the mainstream right is basically apathetic, and rightfully so. Large segments of the left are all energized to do something, and of course the white nationalists who love the Confederacy are angry. The results of this poll indicating that a huge percentage of the population does not favor removing the statues do not surprise me; these people aren't the kinds of folks who are going to care enough to speak up about it.