Non-Gallup/Rasmussen tracking polls thread

[Alcon]

Well, since you've declined to respond to a few ... .

What posts were those?  It was not intentional.  I will respond to any post you link me to.

I would appreciate the same in return.  Here is the post in question.

If you say it's "fairly clear that it [wasn't] methodology," you're saying it's fairly clear that it was a one-in-many-million event.  It must be one, the other, or an accurate observation.  I am still waiting for you to pick one of the three.  They are the only three options.

[Alcon]

Your infamous claims of "evolution is proven."  I never received an answer. 

I don't know how "infamous" it is, but I found the topic again.  I never claimed that "evolution is proven" anywhere in that topic; if I did, I should not have, it's a theory.  Our debate was over whether the development of metacognition was an aberrant pattern that indicated intelligent design.  In the last post I made, I conceded that I didn't really have any way of proving my thesis, or you, yours, but that I thought you were looking for evidence in your own conclusions.

Looking over your last post, you just re-iterated points you had already made.  To me, it looks like that was a mutual agreement to close the subject with mutual disagreement.  You didn't ask any question in the last post that I left "hanging," other than ones which I answered but that you had different interpretations on.

Now, since there are "a few" of these, can you please give me another example?

I am saying, however, that at a subsample really is not indicative of the rest of the poll.

First off, that's factually incorrect.  As Verily pointed out, a swing to Obama 60-40 (closer to in line with national polling) changes the topline significantly.

Second off, you have still failed to pick one of the following options.  Again, one of the following must be true.  Which one do you believe is most probable?

1. The sample is accurate; all other polls are significantly off.

2. The sample is off by random chance, but with no methodology flaw; an event of infinitesimal likelihood occurred.

3. The sample is off by design, and the methodology is flawed, thus the poll is questionable.

"The subsample really is not indicative of the rest of the poll" is invalid because:

1. It affects the topline significantly; and,

2. It shows that their methodology was flawed.

I await your answer -- again, option one, two, or three.

[Alcon]

Well, since you've declined to respond to a few ... .

What posts were those?  It was not intentional.  I will respond to any post you link me to.

I would appreciate the same in return.  Here is the post in question.

If you say it's "fairly clear that it [wasn't] methodology," you're saying it's fairly clear that it was a one-in-many-million event.  It must be one, the other, or an accurate observation.  I am still waiting for you to pick one of the three.  They are the only three options.

"one=in=many-million"?!?

Go back and reread basic survey research methodology.

One out of twenty times the MoE is likely to be exceeded for a sample.

As to subsamples, the MoE is generally so large, its not worth examing.

Go back and read this thread.  The chance that the sub-sample would be off by ~70 points is one in many million, and that is accounting for the high MoE.

I didn't say that the chance it would be out of MoE is one in several million; obviously, that's 1-in-20.  That's the definition of Margin of Error (at the 95th confidence rate, per survey standards.)

[Alcon]

J. J., your option four is not mutually exclusive to the other three.  It demands one of the other three be true.

No matter whether the subsample reflects the poll or not, it must be one of the following:

I. Correct
II. Incorrect, by random chance
III. Incorrect, by poll design flaw

Do you think (I) is likely?  I doubt it.

So, the question is:  Do you think either (II) or (III) is more likely?

[Alcon]

The question was on the ability, basically, or another species to "ask wny."  You never posted the information.

I speculated, just not to your satisfaction.  I posited it was a natural process of evolutionary processes that correlated with evolutionarily-beneficial thinking skills.  You disagreed.  We both pretty much admitted we weren't familiar enough with the evidence to argue fluently.  Again, at that juncture, I assumed we were "agreeing to disagree."

I'm happy to continue discussing this via PM.  If you're still interested, hit me up.  Otherwise, I don't think we should clog the topic.  In any case, if that's the closest I've ever come to dodging a question, I really think I have a damn good record here and you should answer my query without the snark.  You can, of course, agree to disagree.

[Alcon]

Or that the subsample selection was bad, i.e. unrepresentative.  The question is, does a bad subsample knock the poll out of MOE?  I say, with TIPP so far, no.

As Verily posted (which you've ignored three times), it affects the result by more than a couple of points, from Obama +1 to Obama +8.  I haven't checked his math thoroughly, but it appears sound.

The point is not the subsample's effect on the topline.  It is the cause of the bad subsample.  It must either be a random flaw, or sampling error.  In this case,we've already demonstrated that a random flaw has a ridiculously low probability.  Yet, you appear to be totally unwilling to admit that it appears to be a sampling error.  Do you really think a sampling error is less likely than a one-in-many-million event?

If it is likely a sampling error, it impeaches the methodology of the poll.  The methodology was not unique for that one subsample.  The entire poll comes into question.

[Alcon]

J. J., your option four is not mutually exclusive to the other three.  It demands one of the other three be true.

No matter whether the subsample reflects the poll or not, it must be one of the following:

I. Correct
II. Incorrect, by random chance
III. Incorrect, by poll design flaw

Do you think (I) is likely?  I doubt it.

So, the question is:  Do you think either (II) or (III) is more likely?

I think I've said that if it would be III (and I suspect II) it isn't enough to effect the results.  Why.  Because we had similar results in the subsample and the poll behaved the same as the other major polls.

You suspect the chances of flawed methodology are less than 1 in many millions?  Why?

Also, Verily already demonstrated that it would cause a seven-point swing.  It also impeaches the credibility of the poll if it's (III).

Now, convince me that (II) is vastly more likely, and I'll trust it.  I still will not trust the top-line, though, for obvious reasons -- because the "off" sample affects it by many points.

[Alcon]

Alcon, first I asked the question if any non Homo species ever asked why things are as they are.  You promised a link, but never answered.  I actually was disappointed.  

Like I said, if you are generally interested in that discussion, I'll happily continue it by PM.  But the point was that I was not consciously ignoring a point you made.

This is a subsample and I'm looking at if it is sufficiently bad to move the total result out of the MOE.  No.

I'll re-post my post, and highlight the parts you didn't reply to in bold.

You suspect the chances of flawed methodology are less than 1 in many millions?  Why?

Also, Verily already demonstrated that it would cause a seven-point swing.  It also impeaches the credibility of the poll if it's (III).

Now, convince me that (II) is vastly more likely, and I'll trust it.  I still will not trust the top-line, though, for obvious reasons -- because the "off" sample affects it by many points.

The MoE on a poll of this size is +/-3%.  Verily's math shows that this would cause a seven-point swing.  In other words...outside of MoE*!   So, I should bold that part of my post, too.

(* - Why are you convinced that it's OK if a sub-sample is ridiculously outside of MoE, as long as the topline is OK?  Each subsample is its own 1-in-20, yes.  That does not mean that a 1-in-many-million is OK, especially if it affects the result by 7 points.)

[Alcon]

Sorry, considering the results TIPP has had in prior elections, I don't see a methodological problem.  Ah, what part of "and I suspect II" do you have a problem with?  I'm questioning if a sample that would produce a bad result in a subsample would produce a bad result in the whole poll (one out of the MOE).

OK.  You're assuming that a statistically infinitesimal chance is more likely than a pollster having a flawed methodology.  Is that an accurate summary?

In any case, if you'd accepted that the subsample is flawed, are you willing to adjust the overall result to match other polls' subsample (i.e., to Obama +8)?  Why not, if not?

[Alcon]

I'll re-post my post, and highlight the parts you didn't reply to in bold.

You suspect the chances of flawed methodology are less than 1 in many millions?  Why?

Also, Verily already demonstrated that it would cause a seven-point swing.  It also impeaches the credibility of the poll if it's (III).

Now, convince me that (II) is vastly more likely, and I'll trust it.  I still will not trust the top-line, though, for obvious reasons -- because the "off" sample affects it by many points.

The MoE on a poll of this size is +/-3%.  Verily's math shows that this would cause a seven-point swing.  In other words...outside of MoE*!   So, I should bold that part of my post, too.

(* - Why are you convinced that it's OK if a sub-sample is ridiculously outside of MoE, as long as the topline is OK?  Each subsample is its own 1-in-20, yes.  That does not mean that a 1-in-many-million is OK, especially if it affects the result by 7 points.)

[Alcon]

Alcon, did it occur to you that it might overpoll for Obama in other subsamples, but that the average might still be correct?  For some reason, this poll has an excellent track record.  I try not looking too strongly at the subsamples but at the whole poll.

So, you're now arguing their methodology was flawed?  Which one is it?

I'm not looking "too strongly" into them, unless you consider the fundamental laws of statistics to be "too strong."

[Alcon]

J.J. is saying that it all averages out Alcon.  

So, yes, this mistake was just because of randomization.

He believes in the  0.000000002%

True!

Actually, I think he's arguing that not only did the 0.000000002% chance happened, it was probably counter-balanced with another 0.000000002%.

For the record, assuming it wasn't a methodology flaw, that brings us to about 1 in 250,000,000,000,000,000,000,000*.  That's 250 quadrillion, I believe.  That, apparently, is "probably."

* - Numbers for demonstration/hyperbolic absurdity, only, but the real probability ain't far off that.  Maybe even within margin of error!

[Alcon]

Something made that poll the most accurate one in 2004. 

And its performance in 2004, whether it be by solid methodology then, or luck, does not make the statistics here any less damning.

It maybe decreases the probability of "methodology error," but to 0.000000002% (from now on an inexact placeholder for whatever confidence rate is involved here)?  Moreover, to the 1 in 250 quadrillion necessary for another sample to be equally off (ditto caveat)?

No.

[Alcon]

Sorry, but I go by track record.  Now, I'm not seeing this poll, in general being out of line with Rasmussen or Gallup.

In other words, you think that the chance that their 2004 track record isn't representative is greater than "1-in-250 quadrillion" (or whatever the ridiculously high number is)?

[Alcon]

Zogby did pretty good in 2000.

...And I bet they didn't have such a high error rate on their samples.

What say ye, J. J.?

[Alcon]

So, how many times has lightning struck you this year, J. J.?

[Alcon]

To add to Lunar's point, let me re-emphasize:

Subsamples are smaller.  But math accounts for that in calculating their margin of error.  So, you cannot argue that this is fine because of the sample size.  We are taking into account the sample size when we tell you how infinitesimal the chances of it being non-methodological are.

[Alcon]

Or we're taking a subsample that is one of those out of the MOE, when it is independent.

If you are looking at a poll with 5 subgroups, and treating each as an independent poll, which is what you're doing, each subgroup has a 1 in 20 chance of being "bad."  There is now a 5 in 20 chance that one of those subgroups is bad.  That one subgroup doesn't necessarily move the entire poll out of the MOE.  You have to look at the whole poll, not just the subgroups.

You're right...but the chance is still 1 in 2.5 quadrillion or so that two sub-samples would be off, that much, in the exact polar ways.

The thing is, we're not dealing with 1/20.  We're dealing with 1/many million.

[Alcon]

muon,

Fair point.  If it's an unweighted sample, would we have the right idea?  My math education doesn't go much beyond Algebra II so I figured I'd check.

[Alcon]

I'm a little less worried about the volatility (or lack thereof), and a little more about the seeming near-lack of correlation with the other polls.  That's a little odd.

[Alcon]

http://www.pollster.com/blogs/2008%20New%20Hampshire%20Polling%20Methodology.pdf

This?

I'd be surprised if Gallup actually quizzes people.  That seems like it might provoke frustrated/embarrassed hang-ups.

[Alcon]

Average of Rasmussen, Hotline, Gallup traditional, and IBD is McCain %+0.2825 today.  How are you viewing this, honestly?

Honestly, I look at LV on Gallup and I pay very close attention to the 'bots.  If they were to drop and Gallup, IBD and possible ABC would drop or hold, I'd say McCain was closing.

I do think Obama's lead in PA may be eroding, though not collapsing. 

Lunar is excluding Gallup-Expanded, and only including those you indicated, including a M+2 on Gallup-Traditional that may simply be rounding.  Seems a bit leap to "I'd say McCain was closing" from McCain +0.28% in one day -- especially since we have no reason to exclude Rasmussen.  Heck, if you excluded Gallup-Traditional, the average would be moving toward Obama.

If it weren't for Gallup-Traditional, would you say that Obama is expanding his lead?

[Alcon]

First of all, Rasmussen is not unchanged.

Second of all, regardless of where those individual polls are going, the composite change is probably less than 0.5% (is it not?).  Yesterday, based on those same standards, I believe Obama had a composite gain of less than 0.5%.

Yes, if you ignored Rasmussen and assumed that any composite change is probably significant, then sure we'd have a change -- but "ignoring Rasmussen" is a laughably random caveat.  Can you prove that your "trend" reaches statistical significance at the 50% confidence level?

[Alcon]

For what it is worth Drudge has a headline saying zogby will show Obama 49.9% McCain 45.1%. That would move McCain up by 1 point from yesterday and Obama up by 0.5% as well.

In other words, no change. Still a 5-point lead for Obama

Dave

Well, +0.5 in margin for McCain...

[Alcon]

IBP/TIPP has, um, 'corrected' their process:

You must be logged in to read this quote.

Clicky

I personally think he's making some amateur pollster mistakes in 'correcting' his poll.  It's a little Zogbyian, but make your own decisions.