The Problem with 538

[Doctor Jack]

The Huffington Post has been doing some stories looking into 538's methodology lately and they have one very interesting thing that they found which they say (and I agree) causes significant questions about its reliability.

According to 538's model, this map have a 7% chance of happening. This, batshit insane never in a million years map occurs 1 in every 14 times that this election occurs




The reason, according to the article, is that 538 assumes that wacky, extremely unlikely scenarios are actually somewhat likely to happen. The difference between 538 and NYT Upshot, Sam Wang, Daily Kos, Huffington Post, Predictwise, etc is that 538 forces the odds of almost impossible outcomes to be inflated. This causes Clinton's odds to drop because suddenly trump is winning Oregon 1 in every 14 election simulations. Its quite an interesting read.

While I love following the prediction markets for this year’s election, the most popular and widely quoted website out there, fivethirtyeight.com, has something tragically wrong with its presidential prediction model. With the same information, 538 is currently predicting a 65 percent chance of a Clinton victory, while HuffPost’s Natalie Jackson and Adam Hooper are projecting a 98 percent chance,[1] and Sam Wang at Princeton Electoral Consortium is predicting a >99 percent chance.[2] What gives?

538 has been all over the map this election. Their model fluctuates, often irrationally, with each new poll that comes in. Underlying this is an overly complex and opaque set of assumptions that are probably too smart for their own good. If you go to the updates page on the 538 website,[3] you can see how each new poll or set of polls moves their probabilities. It first struck me when a poll on October 27th, showing Trump up by 19 points in Idaho,[4] somehow moved the prediction from Clinton with a 84.4% chance to win the presidency down to a 84.2% chance. I know that’s not a big movement, but why is a poll in Idaho that basically confirms the results moving the national race at all?

If you scroll through the updates, you’ll see countless other similar inexplicable movements. On Friday, Roanoke College came out with a poll that had Clinton up by 18 points in Virginia. Guess how that moved the probability of a national Clinton victory? That’s right, down four-tenths of a point, from 64.3 percent to 63.9 percent. Huh?

All of this would be curious, but acceptable, if it weren’t for the output from the model. 538 has done a ton of empirical work to see what factors should influence the probabilities. I am not questioning any specific assumption that 538 makes, from their state-by-state correlations to their use of a t-distribution to create “fat tails” in their probability distributions (i.e., higher likelihood of otherwise obscure events). What I am questioning is 538’s professional competence and responsibility in reality checking the output of their model.

http://www.huffingtonpost.com/entry/whats-wrong-with-538_us_581ffe18e4b0334571e09e74

[zipplewrath]

2. It's statistics

The map you show suggests a 7% chance of happening. That's 7 times out of a hundred. There have only been about 120 elections in the history of the country, fewer than 60 for presidents. And if you want to discuss anything like "modern times" you've got maybe 30 presidential races from which to choose. That isn't really a "statistically significant sample". So 538 has to be careful about excluding low probability events, because there isn't enough basis from which to judge (and by the way, a map worse than that actually happened in the last 40 years).

He's using very similar methods as he used in handicapping. The problem is understanding the results. He was on ABC yesterday and he pointed out something that is lost on many. When the race is 49% to 44 %, there's only 7% of the vote that is "in play" at this late date. The person with 49% "almost" can't lose. However, when it is 44% to 40%, you have a much greater chance that things can "break" against the leader. The probabilities that they are quoting are effected more by that current situation, than by who is leading or trailing.

Nate will be the first to explain the weaknesses, or instabilities in his model. However, it can be difficult to "compensate" for them without introducing bias. He tends to believe it is better to minimize the impact of some data, as oppose to excluding it and potentially causing bias.

[Yonnie3]

3. All we need is 1,000,000 elections so we can be sure whose method works better. nt

[lapfog_1]

4. The real problem is GIGO

or Garbage In - Garbage Out

The polls are not good this year because they discount non-traditional voters. Especially Latinos.

[AgadorSparticus]

5. OUCH!!! DON'T hold back!!! Ooooeeeee!!! No wonder Nate Silvers went on a tweeting tirade

This----> "What I am questioning is 538’s professional competence and responsibility in reality checking the output of their model".

HOLY SHISTZKY!!