Uh, there's a reason "Reagan Democrats" were a thing.
Race played a PART in Democrats voting for Reagan but certainly not all of it. Many were socially conservative (and apparently finally willing to give the pro-business GOP a try) and were attracted to Reagan's foreign policy ... not to mention disillusioned by Carter.
Also, as I said earlier, Reagan's worst region in 1980 was the Deep South, and though he did very well there in 1984, he ALSO did very well everywhere else, too. Not seeing the clear correlation between racists leaving the Democrats and getting Reagan his victories.
While I agree with you that Reagan's natural base of support wasn't really in the South (more like the Southwest, if you consider suburban Southern California to be Southwestern
), he did expand and update Nixon's Southern Strategy with appeals to racist sentiments-though these tended to be coded in the language of "states rights" and "reducing welfare/government spending." Racism need not be explicit or obvious a la George Wallace in the 1960s.
And keep in mind that the Deep South still had a lot of black and poor white voters who remained loyal to Favorite Son Carter (though the latter-I suspect-went heavily for Reagan in 1984). The broad middle and upper classes of Southern whites that were emerging in the growing suburbs of the South were-and are-extremely conservative.
By the end of the 1980s, the Democratic Party was hemorrhaging voters in the South, particularly in presidential elections but also (increasingly) in congressional elections. Even in the 1990s when Arkansas Favorite Son Bill Clinton appealed to a lot of poorer and working class white voters ( in the Upper South and parts of Appalachia), the South continued on its march to the GOP. It would take adopted Texan George W. Bush to seal the deal.
tl; dr the South's shift to the GOP has been a long, complex process. Opposition to the Civil Rights movement certainly played a significant part, as did the rise of the Religious Right (white evangelicals-in the South and elsewhere-becoming more politically involved in Republican politics). Reagan had a key role to play as well at a crucial point in this process-which
did involve (mostly coded) appeals to racism.