Fivethirtyeight.com's performances
       |           

Welcome, Guest. Please login or register.
Did you miss your activation email?
March 29, 2024, 11:03:04 PM
News: Election Simulator 2.0 Released. Senate/Gubernatorial maps, proportional electoral votes, and more - Read more

  Talk Elections
  Presidential Elections - Analysis and Discussion
  U.S. Presidential Election Results (Moderator: Dereich)
  Fivethirtyeight.com's performances
« previous next »
Pages: 1 [2]
Poll
Question: Do you consider fivethirtyeight.com as a reliable electoral projections website
#1
The most reliable
#2
Very reliable
#3
Quite reliable
#4
Not reliable
#5
The worst
Show Pie Chart
Partisan results


Author Topic: Fivethirtyeight.com's performances  (Read 9560 times)
Antonio the Sixth
Antonio V
Atlas Institution
*****
Posts: 57,968
United States


Political Matrix
E: -7.87, S: -3.83

P P
Show only this user's posts in this thread
« Reply #25 on: January 22, 2011, 07:08:55 AM »

Since the official rsults have now been published everywhere except in Alaska, here is the map showing 538.com's performance for Senatorial races.



The map indicates which party overperformed and by how many points. Grey means the difference between the forecast and the actual results is inferior to 1%. The 30% shade means between 1 and 2.5%, then between 2.5-5, 5-7.5, 7.5-10, and finally over 10%.


Interesting. It appears that Nate projected most races as closer than they actually were.

Yeah. Actually, it was already the case (even more so) in the 2008 presidentials. Every State where dems were underestimated voted for Obama, and every State where democrats were overestimated voted McCain except Iowa.
Logged
Nichlemn
Jr. Member
***
Posts: 1,920


Show only this user's posts in this thread
« Reply #26 on: January 22, 2011, 07:37:09 AM »

I think evaluation of prognosticators needs to take more account of early predictions (as I post about on 538 here). If you only focus on your final forecast, what stops from you posting garbage until shortly before election day?

But races can change dramatically until the election day. So if for example you see a big difference between an early forecast and the final results it doesn't necessarily mean 538 was wrong, just that things have changed.

Sure, you have to take that into account. But it's really just another form of uncertainty that exists in other ways right until Election Day (for instance polling errors). Your 20% predictions two years out or two days out should still be right 20% of the time. The big problem for evaluating predictions, early or late, is the small sample size of elections 538 has covered thus far. 
Logged
Antonio the Sixth
Antonio V
Atlas Institution
*****
Posts: 57,968
United States


Political Matrix
E: -7.87, S: -3.83

P P
Show only this user's posts in this thread
« Reply #27 on: January 22, 2011, 07:41:06 AM »

I think evaluation of prognosticators needs to take more account of early predictions (as I post about on 538 here). If you only focus on your final forecast, what stops from you posting garbage until shortly before election day?

But races can change dramatically until the election day. So if for example you see a big difference between an early forecast and the final results it doesn't necessarily mean 538 was wrong, just that things have changed.

Sure, you have to take that into account. But it's really just another form of uncertainty that exists in other ways right until Election Day (for instance polling errors). Your 20% predictions two years out or two days out should still be right 20% of the time. The big problem for evaluating predictions, early or late, is the small sample size of elections 538 has covered thus far. 

Indeed, that's the big problem. I think we will be able to give some meaningful evaluation only after 10 election cycles, ie 20 years. Tongue
Logged
Antonio the Sixth
Antonio V
Atlas Institution
*****
Posts: 57,968
United States


Political Matrix
E: -7.87, S: -3.83

P P
Show only this user's posts in this thread
« Reply #28 on: May 27, 2013, 05:53:18 AM »

I waited a bit to make sure the results I used were the final ones, but here is finally the Presidential map of over/underperformance compared to 538.com forecasts. Here is it! Smiley



I have tweaked the color coding a bit so as to have more shades. The thresholds are 10 (only WV was off by more than 10 points - sigh...), 7.5, 5, and then it's increments of 1 point (in grey States, Nate was off by less than 1 point).

A quite different map from 2008, anyhow. The pattern seems messier, and with a few areas that clearly stand out (appalachia, the inland northwest, CA, AK, the coastal northeast and the Deep South).


Also, here is another interesting map. It says which component of Nate's forecast (red is the polling average, blue is the "fundamentals" model, and green is the final forecast) was the most accurate in predicting the result. It must be noted that the final forecast is an average of the previous two, which means that it can only be more accurate indicator if the two others skew in different directions. That doesn't mean averaging the two factor isn't the best system.



While in many States the fundamentals were a much better predictor of the result in most States, the polling average tends to be more predictive in the swing States - the States where more polls are conducted. Thus, despite appearances, this actually confirms the validity of Nate's system (which gives a greater weigh to the polling average based on the number of polls).


The same maps will be up for the Senate races soon.
Logged
minionofmidas
Lewis Trondheim
Atlas Institution
*****
Posts: 58,207
India


Show only this user's posts in this thread
« Reply #29 on: May 27, 2013, 06:30:21 AM »

This is basically a map of the fact that nobody good polled the safe states in 2012.
Logged
Snowstalker Mk. II
Snowstalker
Atlas Star
*****
Posts: 20,414
Palestinian Territory, Occupied


Political Matrix
E: -7.10, S: -4.35

P P P
Show only this user's posts in this thread
« Reply #30 on: May 27, 2013, 10:14:18 AM »

Yeah, it's honestly disappointing how safe states were looked at even less than usual.
Logged
Antonio the Sixth
Antonio V
Atlas Institution
*****
Posts: 57,968
United States


Political Matrix
E: -7.87, S: -3.83

P P
Show only this user's posts in this thread
« Reply #31 on: May 29, 2013, 05:36:28 AM »

If you are disappointed by the polls' performance for the Presidential race, wait until you see the Senate map.



Also here, there is a clear global tendency. It's impressive to see how democrats almost everywhere did better than expected. Wow.


As for the components.



Here polling averages actually did a bit better, though you really find situations of all kind
Logged
Snowstalker Mk. II
Snowstalker
Atlas Star
*****
Posts: 20,414
Palestinian Territory, Occupied


Political Matrix
E: -7.10, S: -4.35

P P P
Show only this user's posts in this thread
« Reply #32 on: May 29, 2013, 06:03:52 AM »

Clearly FiveThirtyEight has a radical leftist bias.
Logged
Antonio the Sixth
Antonio V
Atlas Institution
*****
Posts: 57,968
United States


Political Matrix
E: -7.87, S: -3.83

P P
Show only this user's posts in this thread
« Reply #33 on: May 29, 2013, 10:46:24 AM »

Also, it's interesting how little correlation there is between the errors between Presidential and Senate races. The two seem almost unconnected.
Logged
Oldiesfreak1854
Atlas Icon
*****
Posts: 13,674
United States


WWW Show only this user's posts in this thread
« Reply #34 on: May 29, 2013, 04:51:28 PM »

Clearly FiveThirtyEight has a radical leftist bias.
Exactly.  But honestly, 2012 was the worst campaign for polling I have ever seen.  I remember seeing numerous polls in the same state with the same sample that showed a spread of anywhere from one point to 10 or 15 points.
Logged
Pages: 1 [2]  
« previous next »
Jump to:  


Login with username, password and session length

Terms of Service - DMCA Agent and Policy - Privacy Policy and Cookies

Powered by SMF 1.1.21 | SMF © 2015, Simple Machines

Page created in 0.045 seconds with 13 queries.