Nate Silver: "Are the Polls Getting Worse?"

[Clio the Leo]

This is the fourth article in a series on the accuracy of polls and polling averages. In the first two installments, we demonstrated that polls have been extremely accurate at forecasting the winners of governors’ and Senate races in recent years — much more so than you might expect based on intuition alone.

<snip>

There is another type of argument, however, that is potentially more troubling. It could be that, irrespective of the character of this political cycle, polling itself is in decline. This is a widely held view among political elites and many polling professionals — and quite a few of the readers of this blog, I might add.

There are some sound theoretical reasons to think that this is indeed the case. We’ll take these up today, in Part IV of the series. Tomorrow, in Part V, we’ll look at what the empirical evidence says — and make an effort to diagnose just how serious these problems are from the standpoint of our forecasting models. Finally, in Part VI, we’ll address some additional concerns related to statistical modeling more generally.

There are several good reasons to be concerned about the state of the polling industry.

1. Response rates to all types of polls are decreasing, as Americans become more aggressive about screening their phone calls. One academic study found roughly a 30 percent decline in survey response rates (from 36 percent to 25 percent) from 1997 to 2003 — although rates of decline were slightly smaller for surveys that used more rigorous methodology. The downward trajectory has almost certainly continued since 2003.

Moreover, the effects are especially large among certain demographic groups, like young voters. Pew has found, for instance, that only 7 percent of the adults who respond to a typical landline poll are 18- to 29-years-old. This compares to 21 percent in 1995, a figure that is far more representative of young adults’ share of the population.

<snip>

2. Many young Americans — and an increasing number of older Americans — rely primarily or entirely on their mobile phones, which many pollsters do not call. About one in four Americans live in cellphone-only households, and that fraction is increasing every year. In addition, another 15 percent of Americans have land lines installed, but rely principally on their cellphones, and many of them rarely or never accept incoming calls on their land lines, especially from strangers.

3. The proliferation of “robopoll” firms like SurveyUSA and Rasmussen Reports may in and of itself be a problem, or may exacerbate the other problems. About 60 percent of the polls in our database this year were conducted by automated polling firms. Some of them have achieved decent results in recent years. Rasmussen Reports and Public Policy Polling, for instance, have somewhat above-average track records, as measured by the accuracy of polls conducted close to Election Day. And SurveyUSA has had a considerably above-average performance.

But, automated polls are also associated with lower response rates. And some of the firms, like Rasmussen Reports, take other types of shortcuts, like conducting all of their polling in a single evening. Also, as we have mentioned, they almost never include cellphones in their samples. Therefore, it is open to question whether these firms can continue to perform on par with traditional pollsters.

<snip>

All of these problems boil down, more or less, to response rates. Just which fraction of Americans are both able and willing to participate in a survey? And are the Americans who are taking part in surveys representative of those who don’t?

Much more....
http://fivethirtyeight.blogs.nytimes.com/2010/10/04/the-uncanny-accuracy-of-polling-averages-part-iv-are-the-polls-getting-worse/#more-1713

And, the most important question, how many of the people in this photo are sitting at home by their land lines from 5-7pm waiting to eagerly talk to a pollster?

US President Barack Obama shakes hands after addressing a Democratic National Committee Rally attended by 17,000 people on September 28, 2010 at the University of Wisconsin in Madison, Wisconsin.

[ChairmanAgnostic]

1. golly, they may have to change how they poll

ie, going door to door, stopping people in the street, or calling cell phones.

[DURHAM D]

2. Saturday I was outside working in my yard.

About 7:00 pm I looked at my land line and saw that I had missed a call from Gallup in mid-afternoon. They didn't leave a message. I would of loved to talk with them. Hope they call again. That was a first.

[tularetom]

3. IMO the credibility of polls will take a huge dump after Nov. 2

I think the liberal media are in for a big surprise and I predict much backtracking and sputtering on Nov 3.

[Proud Liberal Dem]

8. But it will be oh so sweet

The Repubs especially will look like epic failures after so much "drape-measuring". :evilgrin:

[depakid]

16. Looks like Nate is seting the stage for an interesting post mortem

and if Americans are fortunate, some long overdue accountability among the polling companies.

The take home message thus far:

* For a variety of reasons, response rates are declining throughout the survey industry. This has the potential to create problems, including both reduced accuracy and the introduction of bias.

(Whether the bias is "reliably" systemic and skews the results one way or another might be seen in the aftermath of the election).

* These problems are made worse by firms that take various other types of shortcuts, usually in the name of cost containment.

(This is why cheap media polls differ from and are less accurate than much more expensive internals)

* Pollsters have various defense mechanisms to combat low response rates — particularly weighting based on demographic and sometimes political variables.

(As Nate notes: these methodologies aren't transparent- although should be -if not voluntarily, then through some basic disclosure legislation).

* There is some evidence that pollsters do not behave independently from one another — that the results obtained by one may influence others. For an individual pollster, this may be a defense mechanism of sorts — a poor pollsters can look toward a stronger one for guidance. But it may create additional risks for forecasters if the “consensus” view is wrong.

(See the point above).

In Part V, we’ll look at whether pollsters are winning or losing their battle against declining response rates. Are there signs that the accuracy of polling has begun to deteriorate?

Stay tuned.

It's long been my contention here and elsewhere that polling in America is more often undertaken and used to influence public opinion, not reflect it.

Good to see that )at least in this self interested circumstance) people aren't treating poll results as the Gospel truth as they're all too often eager to do in other situations.

[Tarheel_Dem]

4. Gosh. You mean the crap we've been fed for the past year, might be crap?

Oh, the horror. :wow:

[Proud Liberal Dem]

5. So, if the polls we have to look at now are looking better for us than they were before

then it's entirely possible we are doing even better and may not be in quite as much trouble as most people seem to believe? :shrug:

[kirby]

6. I never take polls...

All my phones are unlisted and if anyone I do not know calls I tell them no thanks and hang up. So obviously I have never been represented in a poll.

[FrenchieCat]

7. So the number of folks attending the Wisconsin rally went down by 10,000???

When did this happen? :shrug:

[Clio the Leo]

9. lol huh? nt

[onehandle]

10. 17k were inside the Library Mall area, the other 10k were in other areas.

Captions are not a threat.

http://www.news.wisc.edu/18465

[elleng]

11. 'Technology' still does it, like 'first' polls,

taken when only 'wealthy' had phones!

'rely primarily or entirely on their mobile phones, which many pollsters do not call. About one in four Americans live in cellphone-only households, and that fraction is increasing every year. In addition, another 15 percent of Americans have land lines installed, but rely principally on their cellphones, and many of them rarely or never accept incoming calls on their land lines, especially from strangers.'

So, polls becoming more and more inaccurate, as 'phones' become more and more obsolete.

[Bluenorthwest]

12. Nate explaining why he is so wrong so often

Frankly, I find him to be simplistic and narrow of vision. He has 'faith' in numbers, but numbers are just one ever changing element in politics. Basically he is a person who claims to predict the future, so he will always have to cover his ass when he is very wrong, as he has been this cycle.

[depakid]

17. Actually in point of fact he's been pretty accurate

and most reasonable people prefer thoughtful analysis like this- along with models setting out probabilities to conspiracy theory and wishful thinking.

[DemocratSinceBirth]

19. Could You Please Cite The Specific Instances Where He Was Wrong

And then what percentage these percentages represent of his total predictions.

Thank you.

[stopbush]

13. Back in 2008, my family of 4 all had cell phones and a landline.

Now, we still have the same thing - except that our land line isn't connected to a phone anymore. I only have the land line because I get a better overall deal on my AT&T Universe account which includes the land line, 4 cell phones, TV and high-speed internet. The $10 a month I spend on the land line saves me over $100 a month in data plans, etc. We don't use the land line. I have no idea whether I get any attempted calls on it. I assume the only people calling it are salesmen, bill collectors and - apparently - pollsters.

More and more families are going to the cell-only model. Even at my work place, I get most of my biz calls on my personal cell, rather than through the company phone system. It's just the way I prefer to have things set up.

I've disconnected my fax machine at home. Who sends a fax these days? Who uses a land line? I only keep it in case the electric goes out and I'm forced to hook a phone up to it to make a call.

If it's true that polling companies haven't adjusted their methods, then it stands to reason that the people they are reaching are going to be older people who sit at home answering their land lines, and that's a demographic that skews Republican.

Which means their polls are worthless.

Big surprise in store for the media next month as the Ds clobber the Rs for the third election cycle in a row.

[Clio the Leo]

14. I dont know of any of my contemporaries who have a land line...

.... and I'm 37.

Also think it's odd how the 'rents will answer their phones (they also have no land line anymore) whether they know the number or not .... I wont answer it half the time even when I know who's calling me. lol

[marlakay]

15. My whole family except 83 yr old mom is cell only

and that means just like me we don't answer any unknown numbers and I have never had a message from a poll. My mom who is only landline owner doesn't answer her phone anymore after some weird calls scared her so she screens all calls with answering machine. My daughter put her on as a family person and got her a cell to, thats the only way any of us call her so only reason she keeps phone is she has slow cheap dial up for her computer just to send emails to grandkids.

As time progresses and the older ones leave us this country will change big time....I have read most of fox viewers are old...what happens when they all go? I don't know any young people that watch them...

Our number one priority should be to keep the internet free because that is our future of free speech to each other.

[old mark]

18. I have never been polled in my life. I guess it would depend on how I was feeling

when I was called, but usually if I don't recognise the caller's number, I let it go to the answering machine.

mark

[DemocratSinceBirth]

20. If Polling Is So Flawed How Did Pollsters Do Such A Good Job In 08?

http://www.pollster.com/polls/us/08-us-pres-ge-mvo.php?nr=1

Maybe the world has changed since 2008 and what worked then no longer works now but we are safe to assume it hasn't until there is proof it has.