New 538 Pollster Ratings

[Gass3268]

Some of their ratings are going to cause some interesting conversations.

https://twitter.com/NateSilver538/status/1375087342344626181?s=20

https://twitter.com/NateSilver538/status/1375090692180021250?s=20

Here's part of the explanation as to why Quinnipiac is still a A-

https://twitter.com/baseballot/status/1375087149914198023?s=20

[GeorgiaModerate]

They also have a related article, https://fivethirtyeight.com/features/the-death-of-polling-is-greatly-exaggerated/, which has interesting data suggesting that live-caller polls are no longer the gold standard.

[VAR]

538 doesn’t take into account polls that weren’t conducted within 21 days of the election, which is perhaps the biggest reason Quinnipiac dodged a bullet.. again. Their most egregious ‘polls’ (e.g. Trump +1 in SC, Biden +21 in ME, Biden +11 in FL) didn’t have an effect on the rating, even though polling was very stable in 2020. They should’ve been held accountable for their clownery.

Anyway... this is disappointing, but not surprising. Let’s just hope Atlas doesn’t eat it up.

[Alben Barkley]

Trafalgar, PPP, and Quinnipiac all have A- ratings LOL

[tagimaucia]

As I expected, AtlasIntel was the best performer in the 2020 cycle.

I just hope their Lula/Bolsonaro polls are as accurate as their US polls were last year! 

[Lief 🗽]

This is really embarrassing. Does anyone actually still take Nate Silver seriously?

[GeorgiaModerate]

A new battle of the Nates!

https://twitter.com/Nate_Cohn/status/1375096889650073606

There's more, click to read the entire thread.

[The Mikado]

Congrats to AtlasIncel. Whichever of you is behind it is doing a better job than any actual pollster.

[Pericles]

538 doesn’t take into account polls that weren’t conducted within 21 days of the election, which is perhaps the biggest reason Quinnipiac dodged a bullet.. again. Their most egregious ‘polls’ (e.g. Trump +1 in SC, Biden +21 in ME, Biden +11 in FL) didn’t have an effect on the rating, even though polling was very stable in 2020. They should’ve been held accountable for their clownery.

Anyway... this is disappointing, but not surprising. Let’s just hope Atlas doesn’t eat it up.

Sure 2020 was stable, but races can easily change in the final 3 weeks (though the huge level of mail-in voting complicates it further). 2016 is a prominent example of this. Even in 2020 though, Biden's lead decreased by 2 points in the last 3 weeks according to the 538 average, so looking back too far could exaggerate the errors. As a general rule then, a 21 day cutoff seems fair.

[Amenhotep Bakari-Sellers]

I  not buying any pollster ratings until next yr, they are betting on another midterm collapse by D's like in previous yrs and the polls are showing the opposite

Just like they have OH and FL safe R and polls are showing D's winning, the midterm jinx is eventually gonna go away when don't have scandal, Trump in 2018 was being impeached

[SnowLabrador]

Throw them all in the trash. I won't take anything Nate Bronze says seriously anymore.

[politics_king]

I'm sure Disney will pull the plug on FiveThirtyEight soon in the next round of layoffs. He's lost a lot of credibility and why I don't want to see people lose their jobs, but this guy has too much sway or at least get him out of there and put in someone to lead the site.

[kwabbit]

Damn, people on here really hate Nate Silver. Faulting 538 for missing in 2016 and 2020 is silly to me. They were still better than every other non-biased predictor/forecaster. You had the jokey Economist model giving Biden massive win probability constantly. They depend off polling data and the polling data was trash. The only polling indicator that the race might be close was Trafalgar, Atlas Intel, Rasmussen, Emerson, etc. And everyone called those polls junk and mocked Silver for not removing them from his model.

I remember when everyone crapped on the 538 model for being too Trump friendly, mocking Silver (Nate Bronze, he needs help, he's lost all credibility). Then after its assumptions were proven more correct than their favorite Economist model, they are now calling it a joke for being too Biden friendly and not predicting the close race.

People are just pissed Biden and the Dems didn't win by as much as they wanted. And they're blaming the forecasters for getting their hopes up. 538 once again did the best job given the crappy polling, but people are just criticizing it because it's the most prominent.

Most of the red avatars will give 538 tons of criticism in 2022 for not agreeing with their hackish takes and then when 538 does better than the red-avatar consensus for the millionth time in a row they'll call 538 a joke.

[lfromnj]

Damn, people on here really hate Nate Silver. Faulting 538 for missing in 2016 and 2020 is silly to me. They were still better than every other non-biased predictor/forecaster. You had the jokey Economist model giving Biden massive win probability constantly. They depend off polling data and the polling data was trash. The only polling indicator that the race might be close was Trafalgar, Atlas Intel, Rasmussen, Emerson, etc. And everyone called those polls junk and mocked Silver for not removing them from his model.

I remember when everyone crapped on the 538 model for being too Trump friendly, mocking Silver (Nate Bronze, he needs help, he's lost all credibility). Then after its assumptions were proven more correct than their favorite Economist model, they are now calling it a joke for being too Biden friendly and not predicting the close race.

People are just pissed Biden and the Dems didn't win by as much as they wanted. And they're blaming the forecasters for getting their hopes up. 538 once again did the best job given the crappy polling, but people are just criticizing it because it's the most prominent.

Most of the red avatars will give 538 tons of criticism in 2022 for not agreeing with their hackish takes and then when 538 does better than the red-avatar consensus for the millionth time in a row they'll call 538 a joke.

The problem is if the polls are trash then they should be graded as such. You can't say your model fails due to bad polling when you think the polling is high quality.

[DaleCooper]

Damn, people on here really hate Nate Silver. Faulting 538 for missing in 2016 and 2020 is silly to me. They were still better than every other non-biased predictor/forecaster. You had the jokey Economist model giving Biden massive win probability constantly. They depend off polling data and the polling data was trash. The only polling indicator that the race might be close was Trafalgar, Atlas Intel, Rasmussen, Emerson, etc. And everyone called those polls junk and mocked Silver for not removing them from his model.

I remember when everyone crapped on the 538 model for being too Trump friendly, mocking Silver (Nate Bronze, he needs help, he's lost all credibility). Then after its assumptions were proven more correct than their favorite Economist model, they are now calling it a joke for being too Biden friendly and not predicting the close race.

People are just pissed Biden and the Dems didn't win by as much as they wanted. And they're blaming the forecasters for getting their hopes up. 538 once again did the best job given the crappy polling, but people are just criticizing it because it's the most prominent.

Most of the red avatars will give 538 tons of criticism in 2022 for not agreeing with their hackish takes and then when 538 does better than the red-avatar consensus for the millionth time in a row they'll call 538 a joke.

I was one of those people that used to give him the benefit of the doubt but after 2020 I've learned to stop taking Silver's model seriously. I don't believe in the conspiracy theories that Trump supporters lie to the polls or are refusing to respond once they hear CNN is conducting it or anything like that, but clearly the big name pollsters are not reaching an accurate sample of the electorate. I think the biggest part of it is because these supposedly credible agencies are basing their polls off of responses from the less than 1% of people that actually answer and engage with unknown phone calls. That methodology clearly no longer works and that's not even mentioning the clear ideological bias of a company like PPP. If Nate Silver is not going to rate pollsters based on their inability to correct the errors that led to previous failures, then his work shouldn't be taken seriously.

[Fmr. Gov. NickG]

I don't have a problem with the revised polling ratings.  But the "Death of Polling" article seems off the mark.

Nate is defending the 2020 polls because they did not have historically bad precision.  

They did, however, have historically bad bias.  And bias is a much bigger problem for projects like 538 than precision is.  In fact, aggregation is perfectly designed to address polling imprecision.  The worse the precision of the polls, the more valuable an aggregation site like 538 is.

But aggregation does nothing to address bias.  If all polls tend to be biased in the same direction, an aggregator will also be biased in the same direction by the same amount regardless of how many polls are averaged.   The aggregator just increassd public confidence in bad polling.

And the even more troubling aspect of the bias is that pollsters did a bunch of things in 2020 to address bias observed in 2016, and the polls ended up even more biased than in 2016.  And the observed bias did not even line up with dominant theories for where we should expect bias (e.g. "shy Trump voters"). 

If 538 wants to retain its value, it needs to push back against the notion that polling in 2016 was perfectly adequate.  Instead, it needs to suggest that polling methodology going forward needs to be fundamentally rethought.

My personal impression is that the directions that polling has gone over the past several years needs to be completely reversed.  Polls have tried to correct for lower response rates by (a) using cheaper contact methods; (b) contacting more people; and (c) correcting their sample through rigorous weighting.  

The direction we should be going in is to assemble a smaller sample, but to spend much more money trying to contact each person in the sample.  This might mean live interviews, this might mean money incentives to respond, etc.  But pollsters need to choose a truly random sample at the outset and stick to it.  If they are unable to contact some part of that sample after doing everything they could, they should simply report how many people they failed to contact and how the demographics of that group compared to the demographics of the contacted subsample, without imputing the votes of the uncontacted subsample based on the results of the contacted subsample.