Non-Gallup/Rasmussen tracking polls thread

[Alcon]

J.J. is saying that it all averages out Alcon.  

So, yes, this mistake was just because of randomization.

He believes in the  0.000000002%

True!

Actually, I think he's arguing that not only did the 0.000000002% chance happened, it was probably counter-balanced with another 0.000000002%.

For the record, assuming it wasn't a methodology flaw, that brings us to about 1 in 250,000,000,000,000,000,000,000*.  That's 250 quadrillion, I believe.  That, apparently, is "probably."

* - Numbers for demonstration/hyperbolic absurdity, only, but the real probability ain't far off that.  Maybe even within margin of error!

[Alcon]

Something made that poll the most accurate one in 2004. 

And its performance in 2004, whether it be by solid methodology then, or luck, does not make the statistics here any less damning.

It maybe decreases the probability of "methodology error," but to 0.000000002% (from now on an inexact placeholder for whatever confidence rate is involved here)?  Moreover, to the 1 in 250 quadrillion necessary for another sample to be equally off (ditto caveat)?

No.

[J. J.]

Something made that poll the most accurate one in 2004. 

And its performance in 2004, whether it be by solid methodology then, or luck, does not make the statistics here any less damning.

It maybe decreases the probability of "methodology error," but to 0.000000002% (from now on an inexact placeholder for whatever confidence rate is involved here)?  Moreover, to the 1 in 250 quadrillion necessary for another sample to be equally off (ditto caveat)?

No.

Sorry, but I go by track record.  Now, I'm not seeing this poll, in general being out of line with Rasmussen or Gallup.

[Alcon]

Sorry, but I go by track record.  Now, I'm not seeing this poll, in general being out of line with Rasmussen or Gallup.

In other words, you think that the chance that their 2004 track record isn't representative is greater than "1-in-250 quadrillion" (or whatever the ridiculously high number is)?

[Lunar]

Zogby did pretty good in 2000.

[Alcon]

Zogby did pretty good in 2000.

...And I bet they didn't have such a high error rate on their samples.

What say ye, J. J.?

[Lunar]

J.J. is saying that it all averages out Alcon.  

So, yes, this mistake was just because of randomization.

He believes in the  0.000000002%

Think about it Alcon, let me explain it to you.

Sure, while this poll, assuming it had perfect methodology, would be 99.999999998% not to get this result.

But did you know that atoms are 99.9999999999% empty?  That's only one more 9 than this poll has.  Punch your wall ten times, did your hand go through it?  WHOOAAH you just defied statistics, it's not that hard, these things HAPPEN ALL THE TIME.

Let me explain this better to you Alcon.  You don't seem to get it.  

How many times have you been struck by lightning?  Has it been more than 1,250 times?  I thought it would be.  You'll get struck by lightning about that many times every time a poll showing McCain winning by this margin has good methodology (minus the subsample repitition issue I've mentioned earlier, so more like 700-800 times).

[J. J.]

Zogby did pretty good in 2000.

...And I bet they didn't have such a high error rate on their samples.

What say ye, J. J.?

It's been good over the last two cycles (third in 2000), and it's in line with the other polls, even when you look at these unusual subsamples.

[Alcon]

So, how many times has lightning struck you this year, J. J.?

[I spent the winter writing songs about getting better]

In 2004 for fun, I wrote up a random generator program in Python to randomly generate the electoral map. Of course the map was completely crazy, but some of the states were correct. If I ran the program once a day for 100 days before the election, no doubt would I at some point come up with a map that looked somewhat realistic for the time.

By J. J.'s logic, on that day my methodology for predicting the map is perfectly valid.

[Lunar]

So, how many times has lightning struck you this year, J. J.?

This year?  That's not fair, he has a 5.6% chance of not being struck by lightning this year under his system (~18 strikes a year).

And if something that is 0.000000002% is an acceptable random occurrence in J.J.'s world, 5.6% is twenty-eight million times more acceptable.

[J. J.]

So, how many times has lightning struck you this year, J. J.?

This year?  That's not fair, he has a 5.6% chance of not being struck by lightning this year under his system (~18 strikes a year).

And if something that is 0.000000002% is an acceptable random occurrence in J.J.'s world, 5.6% is twenty-eight million times more acceptable.

I don't expect to get struck by lightening underground, nor do I expect a subsample to be particularly good.

[Lunar]

Subsamples have to abide by the same laws of statistics as the overall sample.  Subsamples are not immune from criticism.

They are, however, numerous (creating higher likelihood of error) and with a higher MoE, but this result clearly shows that the poll has flawed methodology or is a once-in-a-century type of poll. I don't think you would be defending the poll so aggressively if it wasn't so favorable to McCain (and you could critique me on the inverse, but it doesn't make me wrong).

[J. J.]

Subsamples have to abide by the same laws of statistics as the overall sample.  Subsamples are not immune from criticism.

If this just a pole of the 18-24 year olds, I'd probably agree.  Newsflash, it isn't.

You must be logged in to read this quote.

No, I "defend" the poll because of it's previous accuracy.  I'm not defending Battleground which is more pro McCain; I actually think that one suffers from the R2Kos problem, but in reverse.  I was "defending" TIPP when it showed 7 point lead for Obama.

I'm not a fan of looking at subsamples that are exceedingly small, and trying to extrapolate from there.

I'm also looking at this poll, and trying to see how much different it is from the rest.  Answer is, not a lot.

[Lunar]

Subsamples have to abide by the same laws of statistics as the overall sample.  Subsamples are not immune from criticism.

If this just a pole of the 18-24 year olds, I'd probably agree.  Newsflash, it isn't.

If you've paid attention, newsflash, I'm the only guy that defends you somewhat when it comes to this issue.  Polls with dozens and dozens of samples make outliers d
ozens and dozens of times more likely.

it *IS UNFAIR* to pick through a poll and find things that seem a little off to validate pre-conceived beliefs about the poll.  Every poll with have something a little off.  No poll with a statistically sound methodology will have McCain leading the 18-24 year olds like this poll has, given the 100 person sample size.  Well, maybe one poll every thousand years.

I'm not a fan at looking at subsamples either.  But if they show something that is basically impossible for them to show, it reveals a methodological weakness.  If McCain was winning 5 of the 8 black people (63%) in an Iowa poll or whatever, that'd be no big deal.  But we can look at sample size and account for the normal error that is involved with looking at subsamples and adjust our criticism to account for the reasons why we shouldn't critique subsamples under most situations.  

I agree with you that, in general, criticizing subsamples is bad.  Of course they average out in any good poll and some in every poll with appear off.

But this poll is one in a gazillion or it has flawed methodology.  You can't dismiss this fact under the above rule-of-thumb.  Subsamples are generally useless, but rarely they are incredibly revealing, and this is one of those cases.  This is partly why polls publish their subsamples, so we can understand the inner workings of the poll a little bit better.

[Alcon]

To add to Lunar's point, let me re-emphasize:

Subsamples are smaller.  But math accounts for that in calculating their margin of error.  So, you cannot argue that this is fine because of the sample size.  We are taking into account the sample size when we tell you how infinitesimal the chances of it being non-methodological are.

[J. J.]

To add to Lunar's point, let me re-emphasize:

Subsamples are smaller.  But math accounts for that in calculating their margin of error.  So, you cannot argue that this is fine because of the sample size.  We are taking into account the sample size when we tell you how infinitesimal the chances of it being non-methodological are.

Or we're taking a subsample that is one of those out of the MOE, when it is independent.

If you are looking at a poll with 5 subgroups, and treating each as an independent poll, which is what you're doing, each subgroup has a 1 in 20 chance of being "bad."  There is now a 5 in 20 chance that one of those subgroups is bad.  That one subgroup doesn't necessarily move the entire poll out of the MOE.  You have to look at the whole poll, not just the subgroups.

Now, I keep coming back to the range.  It's about 3.5 points between three tracking polls, with a good record; I think that's less than TIPP's overall MOE.  TIPP might be at the bottom and Rasmussen at the top, so maybe the gap is 5.25.  I only recall one person saying it was below that.

And Lunar, this isn't about defending me.  It's about if this poll is adequate in terms of methodology.  Certainly their track record says it is.

[I spent the winter writing songs about getting better]

Doesn't J. J. frequently argue polls don't matter because they supposedly oversample Democrats or whatever?

[Rowan]

Zogster
Obama 51.1%(-.2)
McCain 41.6%(+.6)

[Tender Branson]

Zogster
Obama 51.1%(-.2)
McCain 41.6%(+.6)

McSurge among Independents ! Obama's lead is down from 26 points to 16 ...

[TheGlobalizer]

Zogster
Obama 51.1%(-.2)
McCain 41.6%(+.6)

MoE +/- 25%

lol

[Alcon]

Or we're taking a subsample that is one of those out of the MOE, when it is independent.

If you are looking at a poll with 5 subgroups, and treating each as an independent poll, which is what you're doing, each subgroup has a 1 in 20 chance of being "bad."  There is now a 5 in 20 chance that one of those subgroups is bad.  That one subgroup doesn't necessarily move the entire poll out of the MOE.  You have to look at the whole poll, not just the subgroups.

You're right...but the chance is still 1 in 2.5 quadrillion or so that two sub-samples would be off, that much, in the exact polar ways.

The thing is, we're not dealing with 1/20.  We're dealing with 1/many million.

[Lunar]

Doesn't J. J. frequently argue polls don't matter because they supposedly oversample Democrats or whatever?

Although he is hypocritical, I think it's fairer to critique weighting samples (where you ignore your SRS and weight  it to pre-conceived beliefs about what the election will actually look like) than randomized selections of subsamples.

I mean, he's still wrong times five trillion, but that's not why.

[Reds4]

I wonder if there is one wacky day's sample in the zogby average and we will see a big shift to Mccain one of the night's soon or not.. heck.. I don't know why I'm even trying to figure it out... it's zogby... the special sauce will likely bounce all over before november 4th.

Zogster
Obama 51.1%(-.2)
McCain 41.6%(+.6)

[J. J.]

Or we're taking a subsample that is one of those out of the MOE, when it is independent.

If you are looking at a poll with 5 subgroups, and treating each as an independent poll, which is what you're doing, each subgroup has a 1 in 20 chance of being "bad."  There is now a 5 in 20 chance that one of those subgroups is bad.  That one subgroup doesn't necessarily move the entire poll out of the MOE.  You have to look at the whole poll, not just the subgroups.

You're right...but the chance is still 1 in 2.5 quadrillion or so that two sub-samples would be off, that much, in the exact polar ways.

The thing is, we're not dealing with 1/20.  We're dealing with 1/many million.

They wouldn't have to be.  An MOE underpolling by McCain across the others might account for it.  That's one of the reasons I'm not looking at subsamples.