Well, most Southern conservatives still voted for Carter. I doubt he would have won much bigger without the incident.
Also crazy to think about the "scandals" of the early days now compared Trump. Just like Poppy dared to check his watch during a debate. As has been stated below, this is ridiculous and comes from the lazy assumption that "White Southerner" = "conservative." That cheapens an interesting political history and wreaks of pushing our current political realities on past eras where they simply didn't dominate...
Of "White Southern Democrats" - even among those who were religious or supported segregation in the past - not all were "conservative," which should be ridiculously obvious. Those who were voting for Jimmy Carter in 1976 (let alone the very large minority who stuck with him in 1980) were hardly right-wingers, and it's obvious that Ford won "Southern conservatives" by huge margins. "Southern populists" or "rural Southerners" or any other not-synonym? Perhaps another story.
