Nate Silver: How To Handle An Outlier Poll

[highplainsdem]

https://fivethirtyeight.com/features/how-to-handle-an-outlier-poll/

It’s pretty rare that a pollster calls his own survey an “outlier.” But that’s exactly what happened last week after a Monmouth University poll showed an approximate three-way tie between Bernie Sanders, Elizabeth Warren and Joe Biden. Patrick Murray, director of the Monmouth University Polling Institute — an A-plus-rated pollster according to FiveThirtyEight — issued a statement describing his latest Democratic primary poll as an outlier that diverged from other recent polls of the race. (Indeed, there were quite a few national polls last week, and most of them continue to show Biden in front, with about 30 percent of the vote, and Sanders and Warren in the mid-to-high teens.)

But Murray doesn’t have any real reason to apologize. Outliers are a part of the business. In theory, 1 in 20 polls should fall outside the margin of error as a result of chance alone. One out of 20 might not sound like a lot, but by the time we get to the stretch run of the Democratic primary campaign in January, we’ll be getting literally dozens of new state and national polls every week. Inevitably, some of them are going to be outliers. Not to mention that the margin of error, which traditionally describes sampling error — what you get from surveying only a subset of voters rather than the whole population — is only one of several major sources of error in polls.

What should you do about these seeming outliers? If you’re a pollster, you should follow Monmouth’s lead and publish them!! In fact, printing the occasional expectations-defying result is a sign that a pollster is doing good and honest work. Plus, sometimes those “outliers” turn out to be right. Ann Selzer’s final poll of Iowa’s U.S. Senate race in 2014, which showed Republican Joni Ernst ahead by 7 percentage points over her Democratic opponent, might have looked like an outlier at the time, but it was the only one that came close to approximating her 8.5-point margin of victory there. The small handful of polls that showed Donald Trump leading in Pennsylvania in 2016 look pretty good too, even though most Pennsylanvia polls had Hillary Clinton leading.

-snip-

About 99.99 percent of you reading this right now aren’t actually pollsters, though. So what’s my advice to you as news consumers when you encounter a poll that looks like an outlier?

To a first approximation, the best advice is to toss it into the average. Definitely do not assume that it’s the new normal. You don’t need to read dramatically headlined newspaper articles and watch breathless cable news segments about it. In a race with many polls, any one poll should rarely make all that much news. But you shouldn’t “throw out” the poll either. Instead, it should incrementally affect your priors. In the case of the Monmouth poll last week, for instance, you shouldn’t have assumed that the race had suddenly become a three-way tie, but you should have inched up your estimate of how well Sanders and Warren were doing compared with Biden.

For extra credit, pay attention to sample size. The Monmouth poll surveyed only 298 Democratic voters, which is small even by the standards of primary polls (which often survey fewer voters than general election polls do). Sample size is a complicated topic — as I mentioned, sampling error is only one source of polling error, and it’s not always the most important one. But as a rough rule of thumb, any poll with fewer than about 500 or 600 respondents is substantially more likely to have outlier-ish results because of sampling error than one that surveyed a larger number of voters. And polls with only 300 voters are especially likely to have issues.

So that’s the Polling 101 answer. When you see a poll that looks like an outlier, just throw it into the average. If you want, you can give some consideration to the sample size and the quality of the pollster.

Silver is writing specifically about last week's Monmouth poll.

But this morning we have a new IBD/TIPP poll that shows Biden at 28%, Warren at 24%, and Sanders at only 12%.

Again, apparently another outlier.

And this one also has a small sample size -- only 360 registered voters who are Democrats or lean Democratic. Silver says that any poll with less than 500-600 respondents is more likely to be an outlier than one surveying a larger number of voters. (The general election IBD/TIPP results from that poll had a much larger sample of over 800 voters, and that showed more typical results, with Biden beating Trump by 12 points, Sanders beating him by 4, and Warren and Harris beating him by 3.)

Anyway, I thought this was a useful article today.

[DrToast]

1. Not sure I agree that IBD poll is an outlier

It’s pretty similar to their poll from last month. Only Warren has shown some movement, but nothing so outlandish that I would question the whole poll.

[highplainsdem]

2. It looks like an outlier to me, but we'll see.

[qazplm135]

3. how is that poll an outlier?

Still has Biden up and at almost 30 pts which is within the margin of other polls.

There are other polls that only have Biden up in the middle single digits so no outlier there.

There are other polls that have Warren in second place and close to Biden.

there are other polls that have Sanders 3d.

The only "weird" thing about this poll is Sanders being so far back in third place.

Otherwise, the results aren't that much of an outlier.