Estimating the Cellphone Effect: 2.2 percent (538.com)

[darius15]

Mark Blumenthal has a rundown of the pollsters that are including cellphone numbers in their samples. Apparently, Pew, Gallup, USA Today/Gallup (which I consider a separate survey), CBS/NYT and Time/SRBI have been polling cellphones all year. NBC/WSJ, ABC/Washington Post and the AP/GfK poll have also recently initiated the practice. So too does the Field Poll in California, PPIC, also based in California, and Ann Selzer. There may be some others too but those are the ones that I am aware of.

Let's look at the house effects for these polls -- that is, how much the polls have tended to lean toward one candidate or another. These are fairly straightforward to calculate, via the process described here. Essentially, we take the average result from the poll and compare it to other polls of that state (treating the US as a 'state') after adjusting the result based on the national trendline.

Since ABC, NBC/WSJ and AP/GfK all just recently began using cellphones, we will ignore their data for now. We will also throw out the data from three Internet-based pollsters, Zogby Interactive, Economist/YouGov, and Harris Interactive. This leaves us with a control group of 36 pollsters that have conducted at least three general election polls this year, either at the state or national level.

Pollster n Lean
========= ====
Selzer 5 D +7.8
CBS/NYT 14 D +3.7
Pew 7 D +3.4
Field Poll 4 D +2.8
Time/SRBI 3 D +2.4
USA Today/Gallup 11 D +0.4

Gallup 184 R +0.6
PPIC 4 R +1.3

AVERAGE D +2.3

CONTROL GROUP (36 Pollsters) D +0.1

Six of the eight cellphone-friendly pollsters have had a Democratic (Obama) lean, and in several cases it has been substantial. On average, they had a house effect of Obama +2.3. By comparison, the control group had a house effect of Obama +0.1 (**), so this would imply that including a cellphone sample improves Obama's numbers by 2.2 points. (Or, framed more properly, failing to include cellphones hurts Obama's numbers by approximately 2 points).

The difference is statistically significant at the 95 percent confidence level. Perhaps not coincidentally, Gallup, Pew and ABC/WaPo have each found a cellphone effect of between 1-3 points when they have conducted experiments involving polling with and without a cellphone supplement.

A difference of 2 points may not be a big deal in certain survey applications such as market research, but in polling a tight presidential race it makes a big difference. If I re-run today's numbers but add 2.2 points to Obama's margin in each non-cellphone poll, his win percentage shoots up from 71.5 percent to 78.5 percent, and he goes from 303.1 electoral votes to 318.5. (The difference would be more pronounced still if Obama hadn't already moved ahead of McCain by a decent margin on our projections).



So this is my plea to pollsters: let's get it right. Perhaps the cellphone effect will prove to be a mirage after all, but that's something for the data to determine on its own, rather than the pollster.



(**) Keen observers will wonder why the average house effect is greater than zero. This is because in determining our house effect coefficients, we weight based on how many polls each pollster has conducted. A couple of pollsters that account for a large proportion of our data, like Rasmussen and ARG, have had slight (very slight, but enough to skew the numbers) GOP leans.

http://www.fivethirtyeight.com/

That is big news. If he is right and pollsters are underestimating Obama's actual standings in the polls by 2 percent, that is huge if it shows up on Election Day.

[tosh]

1. Thanks for this, darius15!!

[napi21]

2. I have a problem with that estimate! I have a cell phone, and if I

don't recognize the #, I DON'T ANSWER THE CALL! Am I just weird, or do most people do that? If they do, how the hell could any polster possibly estimate a % for either Party?

[maseman]

3. My research company will leave a VM and a number

Many do not do this as they need to get interviews done and data tabulated for sending off to the client/press. My company will call a few different times and leave VM if necessary.

[dansolo]

4. Wouldn't that mean that the actual difference is even bigger?

I suspect, because of the cost, that people would be less likely to answer an unknown number on their cell phone than on their land-line. Doesn't that suggest that Obama's additional support would be even greater than the 2.2 % difference?

[napi21]

That's whit I think as well! I don't see leaving a VM would make much diff.

There are LOTS of people on landlines who won't answer the phone if they know it's a pollster, and many who answer just hang up!

I wouldn't answer a cell call, but if I received a landline call, I would be HAPPY to respond to a pollster, just so I could tell them how much of an idiot I think McNuts is, but I have a feeling I'm a minority.

[OnionPatch]

11. I'm curious, why do you only answer landline calls?

Is there some reason why you are more likely to answer your landline? I have both, and I don't answer unknown callers on either.

[Teaser]

5. If one assumes that this tendency not to answer

is distributed randomly across the population of cellphone users, it is possible then to get an estimate just by sampling more users. eventually you'll hit your quota.

[napi21]

6. How are they getting enough cell #'s to do a decent sample size?

If I assume you are right, and I am right about many cell users not answering because of the cost, wouldn't that mean they would have to poll many times more cell phones than landlines, and it THAT'S true, where are they getting the cell #'s? I didn't think there was a list or directory anywhere.

[Teaser]

12. Now that is the rub...

athough random digit dialing might help out there.

[Greyskye]

7. I'm just the opposite

I'll answer all of my cell phone calls, but vigorously screen my home phone. If it comes up on caller ID as an unknown, private, or blocked number they can talk to my answering machine. 90% of the time they hang up.

[msallied]

8. I also heavily screen my cell phone.

I do that because only my friends and a few businesses have the number. If it isn't one of them, then I get suspicious that it's a telemarketer and I do not answer it.

[nickgutierrez]

9. That's since been amended.

PPIC has been removed - the pollsters informed Nate that they haven't been using cellphones for the Presidential cycle. The effect is now calculated as being a 2.8% lean toward Obama.

[Stuart G]

10. please add l.0 to that number (about 3.0 total.) because....

my guess is that l in 100 have no phone...Those no phoners, if we can get to them, are very poor..and very likely to vote for Obama..
The no phoners could make the difference...Please do not sell them short...most people have done that already..Know any no phoners who need to be registered?

[democrattotheend]

13. Gallup Tracking polls cell phone voters

Does 538 take that into account?