Uh, there's a reason "Reagan Democrats" were a thing.
Race played a PART in Democrats voting for Reagan but certainly not all of it. Many were socially conservative (and apparently finally willing to give the pro-business GOP a try) and were attracted to Reagan's foreign policy ... not to mention disillusioned by Carter.
Also, as I said earlier, Reagan's worst region in 1980 was the Deep South, and though he did very well there in 1984, he ALSO did very well everywhere else, too. Not seeing the clear correlation between racists leaving the Democrats and getting Reagan his victories.