CNN Poll: Obama ahead in all matchups

[emailking]

They actually describe it as "a statistical tie." 

Stupid term.

They can't say with 95% confidence that there's a lead.  That does not mean it's statistically identical to a tie.

(Channeling Mark Blumenthal here.)

What do you think it means to be "statistically identical to a tie"?

The term actually makes sense to me. There is no proper way to interpret the results as indicating one candidate being ahead of the other.

[emailking]

There's no way to indicate that a candidate is ahead at the accepted confidence rate (95%).  That doesn't mean that there isn't a candidate leading the poll. 

In the raw numbers maybe, but I think you misunderstand how statistical confidence intervals work. There's no weighting towards the center of the interval. For example, if it is 53 Obama to 47 McCain with 3% MOE, then it is just as likely (to the 95% confidence level) that they are tied as it is that Obama is up by 6, or by 3 even.

We no more have the statistical power to determine it's a tie than we do that Candidate is up by one.  In fact, it's more likely that Candidate is up by one.

That's not true. If they both fall within the confidence interval, than they are equally likely to the given confidence level.

  Mark "The Mystery Pollster"  Blumenthal put it best:

You must be logged in to read this quote.

I agree with that sentiment and that's why I think it's fine to call it a statistical tie. In fact, to me this is what that term essentially means.

[emailking]

emailking,

This is all interesting (and new to me).  So, a 4% MoE with a tie result, is equally as likely (statistically speaking to be) +4 or -4 as a tie?  That seems surprising to me, and I do not understand how that works.

That is correct. CI's do not have weighting towards the center. By construction, CIs are constructed around the measured value of the results that are found. That doesn't mean the measured value is the most likely value.

This is different from creating a normal distribution around a true (known) average value...like the mean SAT score, and saying that's the most likely SAT scores and the liklihood drops off from there.

In the case of polls we are trying to put an estimate on a specific value...the value that we measured.

I feel guilty for passing on incorrect information...the person who taught me apparently fed me bad information.  Sorry J.J.

No worries.

You have the right idea.  Assuming a normal distribution (which is the standard assumption, and the one by which pollsters determine confidence intervals), the probability density of getting the 1.96*sigma case of +4 (or the 1.96*sigma case of -4) is .1465 times (i.e. ~1/7) as likely as getting the published result of a tie.

That is not correct. Consider that a confidence interval could be centered on a range of values that are impossible to happen in reality.

For a poll, you assume a normal distribution to get the range...not because you think the value you measured is most likely to be the actual result if you could poll everyone.

[emailking]

Emailking,

while I recognize what you say in a technical sense (concerning how statisticians cautiously use those numbers) for practical purposes it can hardly be correct. Fundamentally, the point of making a poll is to be able to say that result A is more likely than result B, based on the fact that A is closer to the poll result than B. This must surely be true even if both A and B are within the given CI. What you could do is to simply adjust the CI to leave one of the values outside. In this case, perhaps a 90% CI would exclude the possibility of McCain being ahead.

Granted, technically speaking we're not working with a normal distribution in the regular intuitive way, but I would still say that Alcon's observation is correct.

Adjusting the confidence level to engineer the desired result sounds nice, but think about we're doing here. Let's say we polled 1000 people and 501 were Obama (or leaning Obama) and 499 were for McCain. Now we could use a 1% confidence interval (or whatever it is) and conclude Obama beats McCain and therefore it's more likely Obama is ahead. And that may all be true. And sure it's more likely he's ahead than not ahead if they achieved true random sampling. The question, by how much? Without knowing more (and without the pollsters knowing more too) this cannot be determined. It may not be much ore likely at all that he's ahead, even if the poll shows him up by 3 points.

When it's outside the MOE, that's when you can say "Yes he's ahead now." 95% is the industry standard for making such a conclusion. Short of that it's just informed specaulation.

Granted the polls themselves suffer from a lot more than just sampling error as the population can't even be determined with accuracy nor is the act of measuring accurate (eg. Bradley effect and such).

[emailking]

But, unless I misunderstand, an Obama +1 result means that Obama is more likely to be ahead, statistically, than McCain?  It may be insufficient to say much of anything, but increasing the sample would statistically be more likely to show an Obama lead than a McCain one?  If that's what Gustaf is getting at...

I would agree with this statement: "If we look at all the polls that have had a candidate about 1 point ahead of the other, then in more than 50% of those cases increasing the sample size significantly would have left unchanged the positioning of the 2 candidates in the raw numbers relative to each other."  The problem is, "more than 50%" might mean 50.01%.

The problem here is induction vs. deduction. On the one hand, you are thinking "If Obama really is ahead of McCain, then it is likely my poll will show him ahead." Knowing this instills in us an inherent bias: "If my poll shows Obama ahead, it is likely he is ahead of McCain." Not necessarily.

[emailking]

Can that not also be calculated using confidence intervals?

Sort of. You can do hypothesis testing to within a given confidence level. In this case the hypothesis would be "Obama is ahead" and you can calculate the probability that you accept the hypothesis if it is wrong and the probability that you reject the hypothesis if it is correct. You can't, though, calculate the probability that Obama is ahead.

Well, "likely" is an arbitray benchmark.  I know a +1 poll indicates that there's (this is numerically inaccurate, I'm sure) 52% chance one candidate is ahead.  The difference between 50% and 52% is still worth mathematical consideration, is it not?

Sure. But I don't think that means it's OK to conclude one candidate is ahead of other. Which brings us back to the terminology of "statistical tie."

[emailking]

As regards your above reply, of course using a 1% CI would be a tad absurd. My point was merely that if the result is valid with, say, completely arbitrarily, a 94% CI [cue JJ and Jfern] it constitutes fairly strong evidence. In fact, I vaguely recall someone a while back pointing out that any poll showing candidate A ahead of candidate B was a reason to believe that candidate A was ahead of candidate B.

Of course a poll only shows one ahead of the other if it's outside the MOE.

A second, only somewhat related point, is that when we have several polls, the average of those polls have a much smaller margin of error. If the average of 10 or 100 polls (assuming the polls are unbiased random samples, etc) is A ahead of B by 2% or 3% that would likely be outside of the MoE.

I would agree that when you have many polls all showing the same thing, that is increased evidence of the conclusion even if individually they are within the MOE. You can actually propagate the MOEs to find out just how much evidence you have. The mathematics of course assumes the population is the same for each poll and that true random selection was employed. Generally this isn't the case. But the sentiment is correct.