Nate Silver: Dear Media, Stop Freaking Out About Donald Trump’s Polls

[emailking]

What value is Silver providing? If he's not more right than other pundits, what's the point of him?

Not everything he posts amounts to a quantitative prediction of anything. Some of it is just discussion of what's going on, and what might happen, and how it could happen. It's usually synthesized in a way that's enjoyable to read, even if most of it isn't something you couldn't find somewhere else.

In terms of actual quantitative predictions, he can't be right or wrong because all he ever does is provide probabilities. If he predicts Walker has a 25% chance of winning the nomination, with others trailing behind, that's not a prediction that Walker will do well, but most people who read his column interpret it as such. You can take the probabilities however you want: seriously, for fun, or not at all. It doesn't really matter in the end since they can't be right or wrong. That's not a copout, it's just math.

He's not the only one making quantitative predictions using poll aggregation, but he was one of the first to do it rigorously.

Sometimes he does make an actual prediction on a result but labels it as subjective, so there's no attempt to be rigorous, he's just acquiescing to the fact that he's a human being with subjective ideas.

Ok, so his columns and language suggested that he thought Trump would probably flame out. So what? Almost everybody else did too. He never made a guarantee. He said we should be cautious about assuming his current lead means he will be the nominee, but he's also quick to point out there isn't enough data to say anything definitively. Primary modeling is a lot harder and less accurate than the general election. He's said this many times before.

I think a lot the hate stems from a general misunderstanding amongst the populace of how statistics works and how it should be interpreted.

[emailking]

Actually, this whole year, his predictions for the nomination have been labeled as subjective odds. He's just trying to make educated guesses, he's not using his model to do it.

And the whole Nate Gospelism thing is that he is better than the media - he can tell when they're going to be wrong and goes against them when they're going to be. This year, all he's done is run with the failed media consensus - he said that Bush and Walker would be top contenders for the nomination, they aren't. He continues to say that Trump will fade shortly, even though all evidence points to Trump making it to at least Super Tuesday - Sure, these are things that Atlas ran with for a time, but we're just commoners - Nate, according to his fans, is supposed to be better than that, and he's not.

Has he ever built a model for the primary before though? I honestly can't remember. I know he's done one off regression analyses here and there, but I thought his heavy duty models were always for general elections. I think he's giving the subjective odds because what else can he do, close up shop until the nominees are decided? He knows that polling has very limited predictive value for the primary even this far out. I don't take his subjective odds as anything more than some guy's opinion, but I don't think he means for them to be taken as anything more than that either.

Finally, while what you say about his statistics is technically correct, that's not how Nate takes it, really. When he gave Brazil the best chance of winning the world cup and then they didn't, he outright called it an error. He didn't say "Hey, well this is supposed to happen 45% of the time, and we're living in the 45%.", he instead admitted that he, or rather, his model, had made an incorrect prediction and acknowledged it as such. In 2012, when he made his famous all 50 states correct prediction, he took in all the praise for it. He didn't say "Well, I can't really be right, I'm just throwing out percentages and we just happen to be in the right universe out of the 1000 I simulated", No, he took in all the praise and worship he could. The only time he's used the "It's just percentages" excuse is when it benefits him greatly - best example is when he gave Heitkamp a >90% chance of losing and she won, his claim was "Hey, this shows how good I am - these low probability things are supposed to happen sometimes - if they never did, I wouldn't be good." - it was a clear desire to win some support out of a prediction he knew was terribly wrong through pure spin.

I definitely think it would be better if he didn't do that, but I think it's probably just comes from having to deal with non-statisticians all the time. I remember columns that would come to something like "well it could be x, but y could also happen, but x could be more likely because...." and he would get excoriated in the comments about hedging his bets.

Also, for the Heitkamp thing, that is just right. If he never had (apparent) errors like that, his models would be flawed. But I get your point that he has inconsistent sometimes in describing how to interpret the results.

[emailking]

I'm not at all saying it's a dodge. I understand the math, and I understand how probability works. I'm just wondering whether there's value in it to justify the reputation he got as some sort of wunderkind.

Maybe not. His columns are still a good read though, and his book was very enjoyable. He's admitted himself that his models are only a little bit more sophisticated than poll averages. And I think he said there's only 4 or 5 states that are hard to predict (in a general).

For me the value is in his numbers based analysis vs. just analysis, which is a dime a dozen.

[emailking]

Fair enough. But, at least on the issue of Bush and Walker, it seems like a convenient cop-out regarding their respective flame-outs to say "I only gave them 1 in 4 odds each."

And what would you like him to say? Yes I totally blew it on Walker?

Suppose Walker won and someone calls him out on it, since he predicted there was a 75% chance that wouldn't happen. What should he say then?

If he predicts Walker has a 25% chance of winning, that means he thinks Walker won't win. So why should we rag on him when that comes to pass?

[emailking]

He should acknowledge where his probabilities were closer than those you could obtain from a prediction market.

Closer to what?

But by that logic you could say he predicted that someone other than everyone would win, right? He thought it was more likely that anybody else would win than that any individual candidate would win.

I understand the math, and I understand what he's doing. I just think that argument is a little disingenuous, is all.

The answer to your questions is yes. Any of them is unlikely to win it (potentially), even the most likely candidate. I understand the idea of it may not be palatable. You just have to take it for what it is I guess.

Has anyone ever done so in Silver's case? (I don't know whether his number of predictions are enough to make this statistically significant, but if so, it would be interesting to see). And it would be particularly interesting to see how accurate they are, say, six months before an election and not just the day before.

I haven't seen anything rigorous. I've seen random internet commenters say they did some calculations along those lines and found things didn't pan out well. They could have been cherry-picking of course.

It would be interesting if someone were to do it though, because it's really the only way to assess how good his models are or aren't.

[emailking]

Yeah. The data is there if someone wants to do it. I'm not sure if the senate models were updated daily, but I think it was weekly or maybe a few times a week. Anyway, with several elections now and all those Senate races, there's enough to do it in a statistically significant manner. It would be a lot of work though.