Accuracy of polls in the post-Trump era

[Scottholes 2.0]

I've spoken to many different people about polls showing Democrats leading in the midterms, but they constantly tell me that 2016 taught them not to believe in polls. Here in WI, for example, Democrats were especially disappointed when Feingold lost an upset to Johnson despite polls showing him leading, and we expected Hillary to win our state because she was the clear lead. Most of my coworkers are so sure that Walker will win, for example, despite polls showing otherwise.

While I understand that polls simply show a snapshot of races, are they right that we shouldn't put much faith into them anymore? I am asking this question after a conversation with a few coworkers of mine.

What's interesting is that polls accurately predicted wins in my state until 2016.

[elcorazon]

no

[GeorgiaModerate]

No, people shouldn't ignore the polls; they should just understand what they actually can and can't tell you.  I wrote something about this on the Congressional Elections board yesterday:

Thank you. I think alot of poll watchers dont really get the concept of MoE(I certainly dont). So neither the Sienna and Monmouth polls are out of left field.

I think that's true.  Most people want answers to simple binary questions like "who's going to win?" or at least "who's ahead right now?"  Political polls don't answer those questions; they measure how many people say they'll vote for a particular candidate, which is not the same thing, for several reasons:

1. What people say may not match who they actually vote for.
2. Margin of error as a result of sampling.
3. Outliers (even with a carefully designed poll, you'll get an outlier outside MoE 5% of the time).

So at best, polls provide somewhat fuzzy estimates of support.  But what we really want to know is who's going to win.  It used to irritate me when people would say pollster X got a race "wrong" when they had candidate A ahead by 2 (with a MoE of say 4), but A lost by 1. Meanwhile, they'd say a poll that had B ahead by 15 was "right", because they're only looking at the binary outcome.

Pollsters are continually trying to refine and improve their methodology, but no matter how good they get, they will never be able to generate precise measurements of the outcome.  This is inherent in the nature of sampling and statistics.  The cliche that "the only poll that matters is the one on Election Day" is actually true -- or at least it's the only poll that's truly accurate.  A poll getting very close to a final result includes an element of luck, no matter how good the pollster is.

It's understandable that people just want a simple answer of who's going to win.  That's human nature, and I'm not immune to it.  All we can really do is try and remember what they can tell us, and what they can't.

[Inmate Trump]

The polls weren't really all that off in 2016.

They showed Trump closing the gap significantly in MI, WI, and PA.  They showed a tight race across the board, and that's exactly what happened.

However, they were indeed off in that they consistently indicated a Clinton win.  Polls cannot take into account a few things.  Russian interference, for instance, which DID happen and which DID have an impact on the race.  These were also the two most hated candidates in recent memory (or ever) for a presidential election.  Trump hated rightfully so based on his own words and actions and policy proposals, and Clinton hated for mostly irrational and illogical reasons that those on the right cannot explain without sounding like morons but hated nonetheless.

It's really no wonder the polls weren't as accurate as in previous elections.  If the Democrats nominate someone who isn't universally hated by the right and feared by voters who don't research and don't know any better, then the polls should be more accurate next time.

However, if Russia develops ways to actually hack voting machines (which they attempted to do in 2016), then Trump wins despite polling and despite everything.

[twenty42]

The polls weren't really all that off in 2016.

They showed Trump closing the gap significantly in MI, WI, and PA.  They showed a tight race across the board, and that's exactly what happened.

However, they were indeed off in that they consistently indicated a Clinton win.  Polls cannot take into account a few things.  Russian interference, for instance, which DID happen and which DID have an impact on the race.  These were also the two most hated candidates in recent memory (or ever) for a presidential election.  Trump hated rightfully so based on his own words and actions and policy proposals, and Clinton hated for mostly irrational and illogical reasons that those on the right cannot explain without sounding like morons but hated nonetheless.

It's really no wonder the polls weren't as accurate as in previous elections.  If the Democrats nominate someone who isn't universally hated by the right and feared by voters who don't research and don't know any better, then the polls should be more accurate next time.

However, if Russia develops ways to actually hack voting machines (which they attempted to do in 2016), then Trump wins despite polling and despite everything.

What a scientific and unbiased post...

[Reaganfan]

This goes back before 2016. I started to notice it around 2013/2014. I recall in 2015, the race for Kentucky Governor between Matt Bevin and Jack Conway showed anything from a close race to a few point lead for Conway. The election results had a 9-point win for Bevin. The same thing happened with Houston, Texas Proposition 1 or the "HERO Act" where polling showed it neck and neck, and it ended up failing 60-39.

I wonder, and maybe this isn't correct, but do you think perhaps polling companies poll urban areas too much and underestimate suburban and rural voters? In other words, if you poll Florida and call 100 people but 45 of them live in Miami and 26 of them live in downtown Orlando, you might get a much more pro-Democratic sample, because you're missing areas like the Panhandle, the I-4 corridor, and Collier and Charlotte Counties.

Needless to say, when Trump approval polls range from 39% to 49%, you can surmise he sits at 45% plus or minus 5%. But that's why I don't take any of that stuff seriously.

It's not that I don't trust the polls, but I think they're just one tool to use.

[J. J.]

I think that's true.  Most people want answers to simple binary questions like "who's going to win?" or at least "who's ahead right now?"  Political polls don't answer those questions; they measure how many people say they'll vote for a particular candidate, which is not the same thing, for several reasons:

1. What people say may not match who they actually vote for.
2. Margin of error as a result of sampling.
3. Outliers (even with a carefully designed poll, you'll get an outlier outside MoE 5% of the time).

At least in WI, you can discount 2 and 3, at least in 2016. RCP average showed a 6.5 point Clinton lead.  That was outside of the MOE for all polls and should have taken care of any outliers. 

BTW, WI was the only unexpected state Trump won where the margin was greater than 3.5 points, using the RCP average.  Generally, the polls were within the MOE. 

[GeorgiaModerate]

I think that's true.  Most people want answers to simple binary questions like "who's going to win?" or at least "who's ahead right now?"  Political polls don't answer those questions; they measure how many people say they'll vote for a particular candidate, which is not the same thing, for several reasons:

1. What people say may not match who they actually vote for.
2. Margin of error as a result of sampling.
3. Outliers (even with a carefully designed poll, you'll get an outlier outside MoE 5% of the time).

At least in WI, you can discount 2 and 3, at least in 2016. RCP average showed a 6.5 point Clinton lead.  That was outside of the MOE for all polls and should have taken care of any outliers. 

BTW, WI was the only unexpected state Trump won where the margin was greater than 3.5 points, using the RCP average.  Generally, the polls were within the MOE. 

Very true.   At best, polls are fuzzy estimates even under the best of circumstances, with perfectly designed polls and samples.  If there's a systemic error that throws them off, that introduces even more error, and generally in a particular direction.  In 2016, most polls underestimated the enthuisasm and turnout of WWC voters, and that probably explains the rest of the difference in WI, as well as underestimates in a number of other states.

The fallacy that the OP's friends may be falling into is thinking that "well, polls underestimated Republicans last time, so they will again."  This is a faulty assumption; systemic errors can work in either direction, as in 2012 when many polls underestimated the Democrats. 

Similarly, some people here seem to believe that the R's are certain to do well this midterm because R's "always" overperform in midterms.   It's true that they have outperformed in the last two midterms, but that's because they've been the non-Presidential party.  That's a trend that's almost always true: the out party -- whichever one it is -- overperforms in midterms, barring exceptional circumstances.  Since there's now a Republican in the White House, this trend should favor D's, not R's, this year.

[Amenhotep Bakari-Sellers]

The Dems won't lose MI,WI and PA this time

[Yank2133]

State polling was off in 2016 and I attribute most of that to low number of polls. However, national polls were fairly accurate though (more accurate then 2012 I believe).

[TML]

Remember that 2016 had many voters undecided until the final stretch right before Election Day. Thus, in hindsight, although most polls showed Hillary in the lead, it was possible for late-breaking voters to tip the election the other way (which, according to exit polls, did indeed happen in states like WI, MI, PA, and FL). On the other hand, 2012 didn't have as many undecided voters in the final stretch, which meant that Romney had less of a chance of winning based on late-breaking voters.

We shouldn't discount polls completely, but we should remember that if (1) no candidate polls at 50% or better, and (2) there are enough undecided and/or third-party voters to possibly tip the election the other way, then we should not be too quick to assume that the candidate in the lead in the polls is on track to win. According to 538's data, in hindsight Hillary should have been considered a slight favorite (but not an overwhelming favorite) based on available polls.

[Amenhotep Bakari-Sellers]

If polls are correct and trend starting happening with WI in 2016, some states will vote opposite of their usual party. Just like Dems are winning FL and GA Gov and losing OH & MD.

[Hammy]

Something to consider with the 2016 polls is you had two horribly unpopular candidates, a third party candidate (which always tend to get more support than reality) and a massive number of undecideds, often well into the 10+% range--I remembered there were several polls in Wisconsin for example that had Clinton +5 but with something like 44% and large undecideds/Johnson numbers, so it's not like the polls showed huge blowouts or anything.

Generally (though there are exceptions) of a candidate is at (or at least very close to) 50% and winning--unless the margin is tiny--the candidate that's ahead will win in reality. But this wasn't the case in any of the polls or aggregates that predicted the wrong winner in 2016.

[Since I'm the mad scientist proclaimed by myself]

Didn't they not weigh by education in 2016?

[RI]

Didn't they not weigh by education in 2016?

Yes, and that was a major issue.

[Xing]

Polls that show results which I like or confirm my gut feelings are accurate, polls which don't are fake news.

That's how most people are going to feel about polls from here on out. What a lot of people don't realize that is that sometimes when polls are wrong, your gut feelings are even more wrong.

[Mr. Morden]

Didn't they not weigh by education in 2016?

There were some that did, but many did not.  The ones that did not tended to be state (rather than national) polls, and that may be why the state polls did a worse job than the national polls.  I guess part of the reasoning was that weighting by education (or not) didn't actually matter that much before 2016.  In 2012, for example, the Obama-Romney margin among college graduates was very close to the margin among non-college graduates.  So the thinking was that there was no need to weight by education.  (oops)

[pbrower2a]

Polls are lagging indicators. They cannot predict electoral collapses that happen after a poll. That is what we can remember from 2016. They cannot predict the effects of electoral shenanigans.