538 On the Cell Effect (We keep hitting controversial polling topics, haha)

[Lunar]

Take it how you will.  I'm not endorsing this.

Mark Blumenthal has a rundown of the pollsters that are including cellphone numbers in their samples. Apparently, Pew, Gallup, USA Today/Gallup (which I consider a separate survey), CBS/NYT and Time/SRBI have been polling cellphones all year. NBC/WSJ, ABC/Washington Post and the AP/GfK poll have also recently initiated the practice. So too does the Field Poll in California and Ann Selzer. There may be some others too but those are the ones that I am aware of. (EDIT: The director of the PPIC survey in California has kindly written to let me know that, while they use a cellphone supplement for some of their public policy surveys, they have not done so thus far this year for their Presidential trial heats. The remainder of this article has been corrected accordingly.

Let's look at the house effects for these polls -- that is, how much the polls have tended to lean toward one candidate or another. These are fairly straightforward to calculate, via the process described here. Essentially, we take the average result from the poll and compare it to other polls of that state (treating the US as a 'state') after adjusting the result based on the national trendline.

Since ABC, NBC/WSJ and AP/GfK all just recently began using cellphones, we will ignore their data for now. We will also throw out the data from three Internet-based pollsters, Zogby Interactive, Economist/YouGov, and Harris Interactive. This leaves us with a control group of 36 37 pollsters that have conducted at least three general election polls this year, either at the state or national level.

Pollster                 n   Lean
=========         ====
Selzer                    5   D +7.8
CBS/NYT               14   D +3.7
Pew                        7   D +3.4
Field Poll                 4   D +2.8
Time/SRBI               3   D +2.4
USA Today/Gallup 11   D +0.4
Gallup                 184   R +0.6

AVERAGE                      D +2.8


CONTROL GROUP (37 Pollsters) D +0.0

Six of the seven cellphone-friendly pollsters have had a Democratic (Obama) lean, and in several cases it has been substantial. On average, they had a house effect of Obama +2.8 +2.3. By comparison, the control group had essentially zero house effect (**), so this would imply that including a cellphone sample improves Obama's numbers by 2.8 points. (Or, framed more properly, failing to include cellphones hurts Obama's numbers by 2-3 points).

The difference is statistically significant at the 95 percent confidence level. Perhaps not coincidentally, Gallup, Pew and ABC/WaPo have each found a cellphone effect of between 1-3 points when they have conducted experiments involving polling with and without a cellphone supplement.

A difference of 2-3 points may not be a big deal in certain survey applications such as market research, but in polling a tight presidential race it makes a big difference. If I re-run today's numbers but add 2.2 points to Obama's margin in each non-cellphone poll, his win percentage shoots up from 71.5 percent to 78.5 percent, and he goes from 303.1 electoral votes to 318.5 (EDIT: I have not changed this part of the analysis in reflection of the new numbers, as it should still get the general point across). The difference would be more pronounced still if Obama hadn't already moved ahead of McCain by a decent margin on our projections.



So this is my plea to pollsters: let's get it right. Perhaps the cellphone effect will prove to be a mirage after all, but that's something for the data to determine on its own, rather than the pollster.

(**) Keen observers will wonder why the average house effect is greater than zero. This is because in determining our house effect coefficients, we weight based on how many polls each pollster has conducted. A couple of pollsters that account for a large proportion of our data, like Rasmussen and ARG, have had slight (very slight, but enough to skew the numbers) GOP leans.

http://www.fivethirtyeight.com/2008/09/estimating-cellphone-effect-22-points.html

[Lunar]

You do realize that Pew, CBS/NYT and Time have always had a historical Democratic lean, even before the invention of the cell phone.

My essential problem.

However, I think we're kidding ourselves if we think that this phenomenon isn't going to show up eventually.  How could it not?

Indeed, and never before have young, educated voters been so enthusiastic.

But Sam's right, which is why I didn't endorse 538's analysis.

[Lunar]

We're talking about people who don't have landlines, States.

If you read the article, it claims that seven out of eight cell-only users have favorable opinions of Obama.

[Lunar]

We're talking about people who don't have landlines, States.

If you read the article, it claims that seven out of eight cell-only users have favorable opinions of Obama.

Yeah, but they are young voters and we know their history of reliability when election time rolls around. So it probably evens out.

Well, yeah, that's the common sense counter-argument that comes out, hah.  The point of the article is that it might be finally having an effect.  As Sam said though, the analysis is suspect.

But, like I said, young educated voters are part of Obama's base vote, far more so than Kerry or Gore.  Take that how you will.

[Lunar]

It would seem to me the question is not if cell phone users vote differently, but rather do they differ from non-cell phone users of similar demographic charateristics.

There is no doubt that the 21 year old cell phone only demographic is substantially different than the 47 year old land land owning demographic, but that is not the question..

Pollsters already ensure that young people are appropriately represented in the sample, and that their income, education, etc matches the totality of their demorgraphic -the isolated variable here is just the cell phones, not age, income, education, etc...

IE is the totality of 23 year old males with just a cell phone intending to vote differently that the totality of 23 year old males who may also have a landline?

Well, I imagine the demographics of cell-only users are disproportionately:
a) College students
b) Wealthy
c) Technologically savvy/hip

One would imagine that this year, far more than other years, these demographic descriptors are part of the Democratic base.  I would say that results WOULD differ from the population as a whole.  FivetThirtyEight, in the original article, claims that seven out of eight cell-only users have a favorable opinion of Obama, but I don't know how they got that statistic.

However, even if voting cell-only users are 10% more Democratic than youngin's in general, that's not really significant enough to affect the vote, assuming the pollster weights by age groups. 

[Lunar]

Sure, the cell-phone affect might affect a modern McGovern.  Maybe he would only lose by 22% instead of 23% if he was running today and polled, hah.

[Lunar]

Sure, the cell-phone affect might affect a modern McGovern.  Maybe he would only lose by 22% instead of 23% if he was running today and polled, hah.

Lunar, honestly until I see it actually affecting the election in practice I can't put my faith in the truth of this assumption.

Of course, I'm not asking you to.  That was just a silly hypothetical since he compared Obama to McGovern.  This entire thread is a hypothetical and abstract.

And just so you know, you won't "see" in affecting the election in practice for at least a decade after it happens.  I mean, you can't just assume because the average on the polls on election day were 1% off Obama's result that it was because of the cell-phone effect any more than you can assume the reverse is due to the Bradley effect.  There are dozens if not more independent variables that can cause polling to be off and this is just one of them.

How can you possibly separate the cell-only effect from a good ground game?  Gallup is trying to, by polling cell-only users and comparing it to the overall youth electorate, but that doesn't prove anything for the results on election day.

[Lunar]

It would seem to me the question is not if cell phone users vote differently, but rather do they differ from non-cell phone users of similar demographic charateristics.

There is no doubt that the 21 year old cell phone only demographic is substantially different than the 47 year old land land owning demographic, but that is not the question..

Pollsters already ensure that young people are appropriately represented in the sample, and that their income, education, etc matches the totality of their demorgraphic -the isolated variable here is just the cell phones, not age, income, education, etc...

IE is the totality of 23 year old males with just a cell phone intending to vote differently that the totality of 23 year old males who may also have a landline?

Well, I imagine the demographics of cell-only users are disproportionately:
a) College students
b) Wealthy
c) Technologically savvy/hip

One would imagine that this year, far more than other years, these demographic descriptors are part of the Democratic base.  I would say that results WOULD differ from the population as a whole.  FivetThirtyEight, in the original article, claims that seven out of eight cell-only users have a favorable opinion of Obama, but I don't know how they got that statistic.

However, even if voting cell-only users are 10% more Democratic than youngin's in general, that's not really significant enough to affect the vote, assuming the pollster weights by age groups. 

Why would they be wealthy? If I had more money I would get myself a land-line in addition to my cell phone...though they are probably more urban.

I don't mean incredibly wealthy, just more wealthy than the population that is land-only and a little more money than the population that is both (linked with being more urban).  Using "urban" as a qualifier instead of wealthy might actually encompass wealthy and capture more of the target demographic, so, yeah, use that instead.