Systematic State-Level Polling Bias

[Antonio the Sixth]

So 538 has their polling and forecast post-mortem up and while I agree with the bottom line that the polls didn't have much of a bias overall, I think the one thing they keep missing is that there are consistent biases in specific states. Upon examining 2022 results, I found that these patterns haven't reversed completely. The big peculiar thing was that swing states had a significant Republican bias this year (even swing states that previously had a huge Democratic bias, like Wisconsin). But aside from that anomaly, which might be explained by the flood of junk Republican polls, the other patterns are similar to previous cycles: notably, the fact that Republicans massively overperform their polls in solidly Republican states (Democrats tend to do the same in a few very Democratic states, but not as consistently).

Overall, I've decided to calculate the average polling miss by state over the past 4 cycles. That gives us 2 presidential, 4 senatorial and 2 gubernatorial rounds of elections. Since the Senatorial and Gubernatorial series don't cover every states, I've imputed missing values using a regression model with the presidential data. This is what the patterns look like:

States where the average polling error is less than 1 point are in grey. Then states where Republicans overperform (ie, there is a Democratic polling bias) by more than 1 are in blue, and those where Dems overperform (GOP polling bias) are in red. As you can see, there is a huge swathe of the country where Republicans beat their polls, sometimes by downright comical margins: over 10 points in WV, Idaho and the Dakotas. Mainly, those are states that rarely if ever have competitive races, so it makes little practical difference. However, they also critically include a lot of Midwestern swing states. This is where Democrats got burned in 2016 and to some extent 2018 and 2020. This is also, crucially, where the polling bias reversed itself in 2022 (Democrats overperformed by 3 points in WI, 4 points in PA and 5 points in MI), but overall the average still tends to favor Republicans there. Time will tell if this is a long-term reversal or a fluke. Conversely, California saw very little polling bias in 2020 and 2022, so its history of Democratic overperformance (which was huge in 2016 and 2018) might be over. Of course, a good chunk of the country, including many newer swing states like GA, AZ and NV, see very little if any polling bias.

Going forward, I will adjust my state-level predictions using these polling bias figures, and we'll see if they help me beat the 538 forecast or not. This year it didn't help much, given the aforementioned overperformance in swing states, but if there's anything 538 taught me is to trust averages over one-time blips.

[wbrocks67]

In examining things, I feel like there needs to be a bigger emphasis on specifics. When you talk about the polling, most people need to realize that it any polling "misses" were mostly from being inundated with garbage GOP stuff.

The *nonpartisan polling* was pretty fantastic overall this year on average. However, there doesn't seem to be as much of a serious understanding among pundits and 538/Cook/etc that they really let the GOP stuff that was clearly trash have way too much of an impact whether on their punditry or their models. 538's model having Oz+1 by the end was possibly one of the most egregious examples when it was based on just about 90% GOP polling and only 1 or 2 nonpartisan polls by the end. There doesn't seem to be as much of a postmortem as to why they let their model be so affected by these polls and whether or not to even include some of these firms in the future (i.e. Patriot Polling, with no background whatsoever)

I would also say it's interesting that some of the usual culprits of polling misfires recently (PA, MI, WI, etc.) were either on point in nonpartisan polling for the most part (PA) or like you said, Dems were actually underrated (MI, WI) which was interesting.

Colorado also now seems like a blue state that also seems to overperform polling. Dems overperformed in both 2020 and 2022 there.

[Amenhotep Bakari-Sellers]

There are enough Blks and Latinos to resurrection of the blue wall because RS don't reach out to 30/100 M of us stuck in Poverty but 20/200M whites are in Poverty because of inherited wealth from Europe we aren't from Africa the ones from Africa 3% of us are middle class, Nigerian HS mate had inheritance wealth from her family, that's why we need Reparations 30 K each of us not 5M
Whites and Migrants can have as many kids 2 pregnancies but adoption because they can afford it

It's the Secular Trifecta Filibuster proof that's in questions that's about OH, NC, FL and TX 413 not 303

All we are asking is millionaire pays the same as Firefighters and Police and they do on Property taxes but not Federal tax thats why they cheat they like Trump pays Property taxes but they can afford it


The question. In 22 wasn't us winning PA or WI it was Ryan Deming's and Beasley just like this time it's Brown, Kunce, Tester and Manchin, Manchin has 45% Approvals he isn't DOA he has a better approval than Vance and Johnson at 37%

The RS failed to win a single Sen D seat since 2018 when they won MO, FL, IN and ND how is Brown, Tester and Manchin DOA

[Pollster]

wbrocks67 is right - it does appear that the traditional, methodologically competent/transparent polling outlets did a good job correcting for error in the 2022 cycle (internally, on the other hand, it appears we overcorrected - but I'm not quite ready to talk more about that with confidence yet). But these outlets also polled way less frequently in 2022 than in previous cycles, so it's possible they just had fewer chances to be wrong. Hard to tell for sure.

Regardless, polling bias is becoming more fluid and I think a model to predict it, while informative and helpful, will be difficult to get right by any metric.

The good news is that polling bias does not affect directional results within a poll and therefore keeps it incredibly useful and valuable for campaigns (and keeps me in a job).

[wbrocks67]

wbrocks67 is right - it does appear that the traditional, methodologically competent/transparent polling outlets did a good job correcting for error in the 2022 cycle (internally, on the other hand, it appears we overcorrected - but I'm not quite ready to talk more about that with confidence yet). But these outlets also polled way less frequently in 2022 than in previous cycles, so it's possible they just had fewer chances to be wrong. Hard to tell for sure.

Regardless, polling bias is becoming more fluid and I think a model to predict it, while informative and helpful, will be difficult to get right by any metric.

The good news is that polling bias does not affect directional results within a poll and therefore keeps it incredibly useful and valuable for campaigns (and keeps me in a job).

Please keep us updated on your findings internally! Do you think it was a systemic thing across the board (i.e. national Dems, Dem groups etc. also overcorrected a little too much?)

There's also another piece to this - in that there seems to be serious polling biases among certain groups like young voters and minorities. Quite a lot of the polling issues across the board this year came from terrible sampling among these groups. Some pollsters did get it right, so it IS possible, even with small sample sizes. But I think a lot of pollsters need to figure out how to reach these groups. If you get a survey back and it has the Republican getting 25-30% of the black vote in a swing state, or tied among 18-29 year olds with the Democrat, something is seriously off. Though, I think many of the GOP pollsters were just making these #s up. The biggest nonpartisan offender I think was Emerson.

[Antonio the Sixth]

In examining things, I feel like there needs to be a bigger emphasis on specifics. When you talk about the polling, most people need to realize that it any polling "misses" were mostly from being inundated with garbage GOP stuff.

The *nonpartisan polling* was pretty fantastic overall this year on average. However, there doesn't seem to be as much of a serious understanding among pundits and 538/Cook/etc that they really let the GOP stuff that was clearly trash have way too much of an impact whether on their punditry or their models. 538's model having Oz+1 by the end was possibly one of the most egregious examples when it was based on just about 90% GOP polling and only 1 or 2 nonpartisan polls by the end. There doesn't seem to be as much of a postmortem as to why they let their model be so affected by these polls and whether or not to even include some of these firms in the future (i.e. Patriot Polling, with no background whatsoever)

I would also say it's interesting that some of the usual culprits of polling misfires recently (PA, MI, WI, etc.) were either on point in nonpartisan polling for the most part (PA) or like you said, Dems were actually underrated (MI, WI) which was interesting.

Colorado also now seems like a blue state that also seems to overperform polling. Dems overperformed in both 2020 and 2022 there.

Yes, it does seem the nonpartisan polls did pretty well this year in swing states. This definitely puts me at ease about the possibility of a long-term problem in this regard. The polling in less-competitive states, though (including states that used to be swingy but are increasingly slipping out of reach, like OH and FL) still does seem to have a significant Democratic bias, even this year. That does seem worth looking at, as it's quite possible we get another Senate race in Ohio or Iowa that looks competitive but turns out not to be.


On another note, one thing I didn't look at yesterday is the volatility in the polling error by state. Even if the average polling error is quite large, there are some states where it bounces around a lot, and as a result we can be as confident that the average we see is meaningful. To account for that, I calculated the standard deviation of the polling error in the 8 election rounds I have.

States in grey had average polling error that represented less than half of the standard deviation - in other words, the polling error fluctuates so much from year to year that there's little evidence of a systematic bias. Those with the 30% shade have polling errors between half a standard deviation and one, and so forth with higher shades. So as you can see, there are many states where we don't have that much confidence of a systematic bias, but you still have a few where the evidence is pretty solid - including Ohio with an average error 2.6 times higher than the standard deviation. I definitely think that those states with dark shades are worth keeping an eye on in future cycles.

[Amenhotep Bakari-Sellers]

We know it's a 303map the difference is wave insurance we held onto WI, PA and MI anyways