Nate Silver: Dear Media, Stop Freaking Out About Donald Trump’s Polls

[mencken]

Let's try an actual statistical analysis here. RCP has been generous enough to provide comparable data from the 2008 and 2012 cycles (which both appeared comparably chaotic at the time). The data indicates that roughly half the time, the candidate ends up receiving as his final caucus total within ~5% of his polling average 70 days prior. The only trouble is that this distribution is decidedly skewed right, so you also have extreme examples like the collapse of Cain and Giuliani and the meteorotic rises of Huckabee and Santorum. In light of this information, it might make more sense to survey the field for frontrunners showing signs of wear from media scrutiny (possibly Carson?) and for ascendant social conservatives (possibly Cruz?). Other than that, I see little reason to believe that polling 70 days out will not be reasonably good estimators of final performance. The polls are going to be a good indicator of the final performance of most candidates, and an atrociously poor indicator for two or three of them.

EDIT: I did 30 dumb random simulations (i.e. not taking into account the plausibility of the outcomes) based on the above described asymmetric distribution of past Iowa caucus performances. Trump won a bit less than half the time, Carson a quarter of the time, Cruz a tenth of the time, and the rest scattering. However, even this is almost certainly too bearish for these candidates, since I guarantee that the chance of Bush, Kasich, Fiorina, or Christie winning Iowa is zero.

[mencken]

And the whole Nate Gospelism thing is that he is better than the media - he can tell when they're going to be wrong and goes against them when they're going to be. This year, all he's done is run with the failed media consensus - he said that Bush and Walker would be top contenders for the nomination, they aren't. He continues to say that Trump will fade shortly, even though all evidence points to Trump making it to at least Super Tuesday - Sure, these are things that Atlas ran with for a time, but we're just commoners - Nate, according to his fans, is supposed to be better than that, and he's not.

As I just said in this thread:

https://uselectionatlas.org/FORUM/index.php?topic=224006.msg4817473#msg4817473

show me where Nate is saying that Trump will fade shortly.  He says he doesn't think that he's going to win the nomination, but explicitly says he's not sure about what stage of the game that's going to happen in, conceding that he could hold on to most/all of his current support into the primaries, and lose the nomination.

As for Bush/Walker...I think you're reading that wrong.  Saying A, B, and C are the strongest contenders for the nomination isn't contradicted when A and B flame out and drop out / fail to win any primaries.  You're not saying that those will be the top three in the end.  You're saying that you think one of them will be the winner, but you don't know which one yet.  The second, third, and fourth place winners in delegates could be candidates X, Y, and Z who you give little chance of winning, and you wouldn't be wrong.  I mean, does anyone dispute that Phil Gramm had a better chance of winning the 1996 GOP nomination than Pat Buchanan?  That's not contradicted by the fact that Buchanan actually won some primaries, while Gramm dropped out after Iowa.  Buchanan never had a real chance at winning the nomination, while Gramm did.

What falsifiable metric do you propose to measure probabilistic predictions then? I initially proposed looking at whether they would have made money on the betting markets, which initially looked good for Silver & Co. until smilio pointed out that they likely would not have held those shares indefinitely.

They issued odds again in late September. If you had bought or sold one share of each candidate as Nate Silver recommended then, when you reevaluated your portfolio in early November, you would have made $22 off buying Rubio, $14 off shorting Bush to 25 (it seems to be implied that their reevaluation of Bush occurred after Bush crossed the 25 mark on October 21), $2 from shorting Huckabee, and $1 from buying Christie to 2. You also would have lost most of your investment in Fiorina (-$6), all of it on Walker (-$4), and lost some more money on Trump (-$6) and Kasich (-$1), for a net profit of $22. But again, from a frequentist perspective, his predictions were no better than a coin toss, and this analysis just gets lucky since he correctly realized that Bush was overpriced.

His most recent subjective odds did not substantively disagree with the betting markets except being more bullish on Rubio and Kasich and more bearish on (of course) Trump. He was 1 for 3 in those.

[mencken]

Silver's appeal has rested on his basing things on numbers. He's supposed to have done something that allows him to supersede punditry. I very much agree that there is a lot of misunderstanding of statistics out there, and that doesn't help. But if all he's saying is, "I can never be wrong because I never put 100% or 0% odds on anything," then there's not really much value to his brand of analysis. It really comes down to poll averaging, plus a little bit of incorporation of fundamentals.

You can check how good someone is at assigning probabilities by looking at their overall track record.  Do events that they predict to be 80% likely happen 80% of the time?  Do the 90% events happen 90% of the time?, etc.  But you can't pick out a single one of those events and "prove" that the probability assigned was wrong after the fact, because there's no objective check for such a thing.  That doesn't mean that you can't laugh at the assigned probability for being too optimistic or pessimistic, based on the information at hand.  But those assessments are subjective.  That's not a dodge by Silver.  That's just a matter of definition.

I don't have a problem with people laughing at Silver's assessment of the race.  I'm just confused about people making things up about what Silver is predicting (suggesting that "this guy isn't going to win the nomination in the end" means "this guy's support is going to evaporate tomorrow"), and then saying that he's already been "proved wrong" on this because of polls with Trump at 30%.  And again, this isn't just about Silver, but all of the "party decides" people.  They are largely saying that Trump isn't going to win.  Not that he's not going to get a lot of votes.  Let's wait until the votes are counted before declaring them as having been "proved wrong".

Fair enough. But, at least on the issue of Bush and Walker, it seems like a convenient cop-out regarding their respective flame-outs to say "I only gave them 1 in 4 odds each."

[mencken]

Fair enough. But, at least on the issue of Bush and Walker, it seems like a convenient cop-out regarding their respective flame-outs to say "I only gave them 1 in 4 odds each."

And what would you like him to say? Yes I totally blew it on Walker?

Suppose Walker won and someone calls him out on it, since he predicted there was a 75% chance that wouldn't happen. What should he say then?

If he predicts Walker has a 25% chance of winning, that means he thinks Walker won't win. So why should we rag on him when that comes to pass?

He should acknowledge where his probabilities were closer than those you could obtain from a prediction market. He was more prescient regarding Bush and Rubio's chances, and less so on most of the other candidates.