Election Prediction philosophy, and an introduction to my model

[JustinSmith]

I've been interested in election prediction for the past 5 years (starting with the 2016 presidential election and its primaries). My first models were chaotic and inconsistent, and pretty terrible overall. I've been refining the process and I think I'm in a much better position now, though certainly with much room still for improvement.

Although I initially was fascinated by what Nate Silver was doing, I've thought a lot more in recent years about having a formalized philosophy to guide the process of building a model and making predictions. In coming up with this philosophy, I've also realized that few, if any, analysts are abiding by it.

First and foremost, the primary goal in making a prediction is to evaluate it afterwards. This means that the prediction must be testable (formally, this concept is called falsifiability). If the prediction is poor, then there needs to be a transparent (i.e. public) examination of that and it should lead to modifications to the model. I have not seen this done to any degree I consider satisfactory.

As an example, making a prediction "Candidate X has a 42% chance of winning the election" is not falsifiable. Regardless of whether or not the candidate wins, it does not validate the predicted chance of winning. That's not to say that it's completely useless to estimate a chance (for example, in evaluating the net balance in the Senate you need to have probabilities for each seat that is up for election).

But recently I have developed a model that focuses more on testable results. It produces an estimated vote total for each candidate. After the election, the prediction can be quantitatively evaluated by measuring the deviation between the prediction and the result, both in terms of total votes and in terms of vote ratio. My model is still very much a work-in-progress. It needs lots of testing and likely a lot of revision. My goal is an average accuracy percent (according to how it's defined in the model) of at least 95%, as early in the process as possible.

Its first test was the California Recall Election recently. This isn't an ideal first test because it's not a normal type of election. My last prediction for this election was:
"Keep"/Newsom: 5,786,811
"Recall": 3,821,253

This election is only 95% counted thus far, so the evaluation is not yet complete, but it's enough for a rough evaluation. Here is how it deviated and what I learned:

1. Far more people voted than the model predicted (it was a record for CA governor race). With 95% of the votes counted, my prediction was 69.83% accurate for vote total, which falls into the range I consider unacceptable. I realized that I didn't have a process for determining election enthusiasm. I predict a total vote by looking at previous elections and using several linear regressions and the 4-point average. Going forward, I'm already deploying an election enthusiasm factor for the VA and NJ elections this November. It's based on the frequency of polls, the theory being that the frequency of polling is related with the enthusiasm in the voter population for participating in the election. Further testing will have to evaluate how accurate this theory is.

2. My ratio was good. I predicted "Keep"/Newsom would win by 20.46 points. With 95% counted, "Keep"/Newsom won by 24.22 points. According to how I've defined the ratio accuracy, this prediction was 92.47% accurate for vote ratio.

The average accuracy for this prediction using the model was 81.15%, which falls between my thresholds for "Acceptable" and "Good". Ideally I want to hit 95% consistently.

I mentioned that I currently disvalue "chance of winning" numbers due to difficulty in testing them afterwards, but for the record, the model had a 100.00% (rounded to nearest 0.01%) confidence of "Keep"/Newsom winning the election. If this number seems abnormally high, you are correct. My model is currently more confident of its predictions than other models tend to be, and that's something I aim for with it. Hopefully as I develop and tweak the model more, it will become more justified in its confidence.

[beesley]

Interesting stuff and welcome to the forum. I look forward to what this will bring.