Reply #99: heavy reading [View All]

[foo_bar]

I found a textbook called "Analyzing Elections", to be published next year. Half of Chapter 10 is about Mitofsky's rise to internet infamy from 1967-2004:

Exit polls were first conducted in 1967 at CBS by Warren Mitofsky. Originally exit polls were used by the statisticians to get a better idea about what to expect in the election, they were seen as guideposts rather than as consequential sources of data.

(37 years later...)

The problems with the exit polls highlighted the difficulty with having one source for exit poll data provided to many clients and the potential for leakage of the results without statistical interpretation. The early results also placed a greater weight on the responses of women, who were more likely to support Kerry. While these weights were corrected later on election day and before the results were used for analysis or projecting outcomes, they increased the popular perception that Kerry was winning the election.

http://www.nyu.edu/gsas/dept/politics/faculty/morton/book/MortonElectChap10.pdf (the manuscript's free until it's in-print)

I guess the professors write the history. But wait, it gets worse:

2. What is the difference between the ratio estimators used by Edison/Mitofsky to project states and the linear estimation procedure used by UNIVAC and Ray Fair? What are the advantages and disadvantages of the two approaches? Explain.

(hint on page 244: "...the engineer in charge of Remington Rand’s new product development division went on camera and apologized to the viewers: “A mistake was made. But the mistake was human. We were wrong and UNIVAC was right. Next time we’ll leave it alone.”")

For a less pedagogic cite, "A Review of Recent Controversies Concerning the 2004 Presidential Election Exit Polls", commissioned by the SSRC (Walter Mebane of butterfly-ballot fame is on their election commission). Compare and contrast:

This information would prove especially important in 2004 when NEP was working with a brand new version of these systems. The data also provide information about estimated point differences between the candidates, and they are often used by the sponsoring organizations to "characterize" Election Day on the early evening news. The news organizations use this NEP information to plan the final allocation of their resources for on-the-air interviews, the last version of the scripting of election night coverage, or the early coverage in newspapers without making any specific calls. But the early data were never intended for general distribution to the public.
...
The data that are deposited are weighted in a complex way that accounts for some nonresponse adjustments, the demographic characteristics of the respondents, and, most importantly, to the actual outcome of the election in a state or to the national outcome. When the outcome of past elections was not as close and the introduction of new voting technology was less common, this was a satisfactory procedure. But in the context of the leaks in 2004, the competing theories about why and how this happened, and the quality of the exit poll results, data weighted to the actual outcome of the votes was no longer a satisfactory dataset for many consumers of exit poll results.

Eleven weeks after the election, NEP and it sponsors released a 77-page report on what happened. This is an unusual analysis in that the same people who were responsible for the exit polls and the projection apparatus also did the evaluation. Furthermore, it is complicated in a way that many post-survey evaluations are by the fact that some information is essentially unknowable. This is especially true when one of the concerns is nonresponse, and there is no information from the nonrespondents to analyze. As a result, there are some sections of the report in which there is an extremely detailed level of disclosure about what the exit poll data show, but in other parts of the report there are only hypotheses about what might have been the cause for a particular observation. These hypotheses can guide future experiments in exit polling methodology or even direct changes in the methods, but they cannot explain in a strict causal sense what happened in the 2004 data collection.

The report actually discusses three issues, two of which received relatively little attention prior to its release. The bulk of the report is devoted to an analysis of how the exit poll interviews were used to estimate the actual support for George W. Bush and John Kerry , and it acknowledges that there was a Democratic bias in the poll. It also contains a discussion of a variety of survey weighting issues, and a statement about technical problems with the computer system that caused two disruptions to operations during the course of the evening. There is an unprecedented amount of information available in the report, much more than has ever been released before. This has prompted considerable discussion of the NEP report on the Web, but it has not silenced the critics of the NEP and their data.

http://elections.ssrc.org/research/ExitPollReport031005.pdf