NYT LIVE POLL THREAD:

[Antonio the Sixth]

Interesting concept.

[Antonio the Sixth]

Even 500 is a pretty small sample, especially if you're going to use weighting which tends to increase the margin of error.

[Antonio the Sixth]

#analysis in this thread is godawful even by Atlas standards.

[Antonio the Sixth]

I hate the way 538 is handling these live polls. They seem to be entering them every day as completely separate data points. That means that KY-06 will likely go into their model as three different polls despite the fact that over a fifth of the voters will be the same throughout. Also, Sienna (the company doing the polling) is saying pretty clearly that, unless the polls have over 500 responses, they are not finished. It seems to me like a poor practice on 538's part.

https://projects.fivethirtyeight.com/polls

I sent feedback to 538 about the KY-06 poll being included with a sample size of 164, and got a response this morning: "...we're updating any Siena/Upshot poll once its sample size is over 150. Then we put in the finished poll once it's done."

This is ridiculous. I can't believe that people at 538 don't know better.

[Antonio the Sixth]

I hate the way 538 is handling these live polls. They seem to be entering them every day as completely separate data points. That means that KY-06 will likely go into their model as three different polls despite the fact that over a fifth of the voters will be the same throughout. Also, Sienna (the company doing the polling) is saying pretty clearly that, unless the polls have over 500 responses, they are not finished. It seems to me like a poor practice on 538's part.

https://projects.fivethirtyeight.com/polls

I sent feedback to 538 about the KY-06 poll being included with a sample size of 164, and got a response this morning: "...we're updating any Siena/Upshot poll once its sample size is over 150. Then we put in the finished poll once it's done."

This is ridiculous. I can't believe that people at 538 don't know better.

Nate Silver is extremely bitter at NYT. It’s just people trying to ingratiate themselves with the boss.

How does entering polls wrong in their database hurt the NYT in any way? It only hurts 538 since it risks skewing their forecasts.

[Antonio the Sixth]

https://twitter.com/Nate_Cohn/status/1038486230168752130

That's actually a good point.

[Antonio the Sixth]

It's funny how fast these change. Last night Paulsen and Radinovich were dominating, and now both races are close.

It's almost as if small sample sizes yield unreliable results...

[Antonio the Sixth]

This thread is a bad idea.

[Antonio the Sixth]

OK, I'm not going to look at that page ever again. I already have enough stupid, pointless things to stress about in relation to the election as it is. I'll look at these polls' results only when 538 adds them to their model (and only to the extent that they actually affect Dem's odds).

[Antonio the Sixth]

Complaining that the sample size for a single CD is around half of what we'd get for a statewide or national poll is... an interesting media strategy.

Like, Nate Silver understands basic error band calculation, right? Is he just being opaquely misleading for his dumber followers?

Anyway, these polls all seem about near expectations so far. I don't know what the big fuss is.

I think Nate's just trying to point out that these CD polls are inherently going to be more error prone than ones from larger jurisdictions typically are.

But that's not how statistical error works. A smaller population naturally demands a smaller sample size for a given error bound. He's being deliberately misleading by implying that the relatively-smaller absolute sample size lessens the significance, unless he's really much less versed in his own industry than we've all assumed. (I won't rule out the latter possibility. He doesn't impress me.)

You're the one who doesn't get how statistical error works. The predictive power of sample size has nothing to do with the size of the target population. A 1000-person poll of the entire world population would be more accurate than a 500-person poll of a mid-sized city, provided that sampling is truly random (of course, sampling is not actually random, but that's equally true of House polls as it is of State polls - and weighting tends to make smaller sample sizes even more noisy if anything).

[Antonio the Sixth]

Complaining that the sample size for a single CD is around half of what we'd get for a statewide or national poll is... an interesting media strategy.

Like, Nate Silver understands basic error band calculation, right? Is he just being opaquely misleading for his dumber followers?

Anyway, these polls all seem about near expectations so far. I don't know what the big fuss is.

I think Nate's just trying to point out that these CD polls are inherently going to be more error prone than ones from larger jurisdictions typically are.

But that's not how statistical error works. A smaller population naturally demands a smaller sample size for a given error bound. He's being deliberately misleading by implying that the relatively-smaller absolute sample size lessens the significance, unless he's really much less versed in his own industry than we've all assumed. (I won't rule out the latter possibility. He doesn't impress me.)

You're the one who doesn't get how statistical error works. The predictive power of sample size has nothing to do with the size of the target population. A 1000-person poll of the entire world population would be more accurate than a 500-person poll of a mid-sized city, provided that sampling is truly random (of course, sampling is not actually random, but that's equally true of House polls as it is of State polls - and weighting tends to make smaller sample sizes even more noisy if anything).

The size of the population does have an effect, just a fairly small one when you get up to large populations. You can use this calculator:

https://americanresearchgroup.com/moe.html

To see a significant effect, compare using relatively small populations.

For example, a sample of 500 with a population of 1,000 is 3.1 MOE.

Whereas a sample of 500 with a population of 10,000 is 4.27 MOE.

https://americanresearchgroup.com/moe.html

What's the theory behind it? That doesn't match what I learned in class about the frequentist approach to stats and the central limit theorem, but maybe pollsters are using a different framework?

[Antonio the Sixth]

What's the theory behind it? That doesn't match what I learned in class about the frequentist approach to stats and the central limit theorem, but maybe pollsters are using a different framework?

https://en.wikipedia.org/wiki/Margin_of_error#Effect_of_population_size

Sometimes an approximation whereby the population is assumed to be the same as though it were infinitely large is used. That is a reasonably good approximation in some circumstances, but only when the population is indeed quite large (see the example numbers I posted above where it makes relatively little difference for large populations, but relatively more difference for small populations)

Frequentist vs. Bayesian methodology wouldn't really make much difference here, I don't think.

Oh, I see. That makes sense. Still, making this assumption could only bias the margin of error upwards, which means it's a fairly reasonable conservative assumption to make. And the gain in precision from estimating a finite population is so small that I'm not sure why pollsters even bother (or why AMB1996 thinks it's so damning that Silver didn't mention it). The effect of the non-randomness of samples, and subsequent need for weighting, is far, far more significant.

[Antonio the Sixth]

Weighting basically makes the MoE into a meaningless statistic. The real MoE to look at would be that of each weighted subsample - which, needless to say, is enormous.

[Antonio the Sixth]

Are we really going to cheer the fact that Hispanics aren't turning out for the sake of a cheap gotcha?

[Antonio the Sixth]

Yeah, I'd definitely say that any poll with a ratio higher than 1.4 should be looked with extreme skepticism. Honestly, I'm even suspicious of all those over 1.2.

[Antonio the Sixth]

So some people on this thread have been disputing the quality of these NYT polls, and I have finally found the solution to figure out how precise each poll is.

...

According to this, the 5 most accurate polls so far were, in order, MN-08, MN-03, WI-01, KS-02, and CO-06.

The 5 least accurate ones, or the ones most weighed, would be from largest to smallest, FL-26, TX-07, WV-03, VA-07, and TX-23.

Precision is not the same thing as accuracy.

Precision is the MOE. The design effect is incorporated in the MOE, so it should not be any great mystery or some surprise if you are interpreting the polls taking into account their MOE, which you should be doing.

Accuracy, on the other hand, is how close to the final result the poll's topline turns out to be (something that is not knowable until the votes are actually tabulated).

A poll can be extremely accurate and not precise at all. And vice versa, a poll can be extremely precise, but not accurate at all.



https://manoa.hawaii.edu/exploringourfluidearth/physical/world-ocean/map-distortion/practices-science-precision-vs-accuracy

You must be logged in to read this quote.

The MoE is not a meaningful statistic in polls that use weighting. The whole theoretical basis for the MoE rests on the idea of random sampling with equal probability.

[Antonio the Sixth]

If they really want to stick to their weighting methodology, then they have to make sure that ever demographic they weight has a sample size of at least 50 (ideally 100, but that would probably be too expensive). Otherwise this is really bullsh*t, yeah.

[Antonio the Sixth]

https://twitter.com/Nate_Cohn/status/1042998029093982210
https://twitter.com/Nate_Cohn/status/1043008295672279041

The obvious solution is to collect a bigger sample, but that would make this whole publicity stunt too expensive I guess.

[Antonio the Sixth]

(tweets about changing poll weights going forward)

The obvious solution is to collect a bigger sample, but that would make this whole publicity stunt too expensive I guess.

Early in the project, there was a Twitter discussion where Cohn talked about this.  Given a finite amount of resources, they preferred to do 100 polls of N=500 rather than 50 of N=1000 in order to maximize the number of districts polled.  For the same reason, they wouldn't be doing Senate polls because those are certain to get a number of polls from other organizations.

I get why they made that choice, but I still think it's a bad one. You can infer a lot more from 50 high-quality polls than from 100 noisy polls that are thrown all over the place by both random and non-random error.

[Antonio the Sixth]

Here's a more useful metric: how each poll compares with 538's "fundamentals" forecast. That gives us a better sense of which of these polls were "good news" or "bad news" for Dems.

WV-03: D+19.3
CO-06: D+6
IL-06: D+5.5
MN-03: D+5.3
NM-02: D+3.8
MN-08: D+3.7
KY-06: D+2
KS-02: D+1.7
WI-01: D+0.2
IL-12: R+1.3
VA-07: R+1.3
CA-49: R+1.9
IA-01: R+3.6
CA-48: R+4.8
NJ-07: R+5
ME-02: R+5.1
FL-26: R+5.6
TX-07: R+6.5
CA-25: R+11.7
TX-23: R+15.2

Mean: R+0.7
Median: R+1.3

So these polls overall have been slightly Republican-friendly, although there are big outliers on both sides.

Although the addition of TX-32 and KS-03 might change that. They're almost done and currently stand at D+10.1 and D+14.7, respectively.

[Antonio the Sixth]

SMALL. SAMPLE. SIZES. GIVE. UNRELIABLE. RESULTS.

Why can't you people get this through your skull ffs?!?!?

[Antonio the Sixth]

Icespear is right.