If there was fraud in 2000, would that have affected exit polls in 2004?

[passy]

If there was electronic miscounting of votes in 2000 which gave more votes to *, wouldn't that have affected the "Who did you vote for in 2000?" question of the exit poll?
Because If I thought that I voted for Gore and my vote was counted for *, I wouldn't be lying to the exit pollsters but my answer would be wrong.
Let's project to 2008 what will happen to the exit poll then, if there was as we are now discovering slowly, massive fraud perpetrated in 2004.
If we can't trust the results of 2004 how can we weight the 2008 exit polls?
I would appreciate any answer even if this topic has been discussed earlier. Thanks.

[CottonBear]

1. Check out the Election Reform forum:

http://www.democraticunderground.com/discuss/duboard.php?az=show_topics&forum=203

[dflprincess]

2. What do you mean "if"?

n/t

[OnTheOtherHand]

3. what you say is true, although...

'massive' electronic vote miscount favoring Bush in 2000 seems unlikely to me, because the pre-election polls generally showed Bush slightly ahead (but Gore closing), and the 2000 exit polls don't point to massive discrepancies either (Florida could be an exception). Of course Gore got more intended votes in Florida, but probably not by a huge percentage -- that's leaving out all the people who never even tried to vote because they got purged.

Also, I wrote a whole paper about why I think we can't trust people's answers to the "Who did you vote for (last time)" question anyway. (Full disclosure: some people apparently think Karl Rove paid me to write it.)

http://inside.bard.edu/~lindeman/too-many.pdf

But that's just one question on the exit poll questionnaire. It doesn't influence what most people think of as "the results," i.e., the vote projections.

[mopinko]

4. exit polls called florida for gore

which, if you can in any way credit gore for the butterfly ballot votes that were trashed, he won in a landslide. that is why the voter news service was destroyed.
but i am not clear on your premise here. what do you mean by weight the polls?

[passy]

6. Well !

One of the problems with the final exit poll was the weighting done on the "Who did you vote for in 2000?" which showed that over 100% of the people who voted for Bush in 2000 voted for him in 2004.
If we assume that * only got 48% of the intended votes but 51% of the counted votes. If people are asked in 2008 "Who did you vote for in 2004?" won't the results be flawed even taking into account the false-recall theory as I think it is called?

[OnTheOtherHand]

8. apart from my position, there's a part of this that isn't controversial

(as far as I know). Everyone agrees that the 2004 exit poll tabulations were weighted to match the 2004 vote returns, and everyone agrees that the resulting "who in 2000" results were "impossible." (Many DUers think this points to massive fraud; as far as I can tell, very few political scientists do.) So, the "who in 2000" results didn't affect the rest of the exit poll results; they were one strange table in their own right.

If there was massive fraud in 2004, and/or if some people misremember who they voted for, then the "Who did you vote for in 2004?" question will be flawed, even if 2008 is clean. But there is no obvious reason why that would affect the rest of the exit poll.

[passy]

16. I don't get your point.

"there is no obvious reason why that would affect the rest of the exit poll"
Well then why did they have to change the numbers for that particular question when they delivered the final exit poll.
Maybe that wasn't the point but when they re-weighed the poll to make it match the results, by increasing the number of people questioned who voted for *, they forgot about those numbers which look ridiculous.
Why did they even bother to make the final exit poll match the results the day after, to make it look like they had messed up less?
Their adjustments make no sense.
If one believes that the election was fraudulent one must understand that the exit poll unwittingly would reflect that.
Quite simply because in order for * to win he would have to have been given more votes than Kerry even though he didn't.
The fraud is sometimes illustrated by precincts with turnouts in the high 90s, third party candidates getting more votes than they should, candidates from the presidential candidate's party obtaining more votes than them and possibly exit polls. I mean even if the methodology was slightly off it seems that exit polling is more accurate than actually voting. It reflects how people thought they voted rather than their actually ballot, which more often than not either doesn't exist in anything other than bit form or might simply be thrown away and never be counted like thousands of provisional ballots.
Just explain to me how flawed an exit poll would have to be to not reflect the extent of the fraud perpetrated in Ohio for which there is no lack of evidence. I think that it is actually harder to make the exit poll not reveal the extent of fraud rather than the other way around. You can try, but then again you come up with stupid numbers that defy logic.

[OnTheOtherHand]

17. umm, you seem to be taking up an argument I'm not

I can explain how the exit poll works without our having to argue about whether the exit poll offers proof of fraud. Does that make sense?

I would be happy to explain why I don't think the "Who did you vote for in 2000?" question proves fraud -- but I wasn't trying to hijack your thread in that way. That certainly wasn't the question you asked in your OP.

My point was the same one that karyn made more succinctly two minutes later: the "who did you" question isn't used to weight anything else. (I didn't mean to be confusing, but I am hot, sleepy, and distracted!)

EDIT to clarify the header

[passy]

19. I'm not arguing just discussing.

I'm not sticking to the OP, sorry. I was just conversing with you on different points.
I didn't think that the "Who did you vote for in 2000?" proves fraud, the exit poll might not prove fraud either it simply indicates fraud.
Like a seismic reading after a North Korean nuclear test. They didn't tell us they carried it out, but we have a clear signal that they did. A seismograph wasn't made to detect a nuclear blast but it does.
I think we can look at the exit poll the same way, it wasn't meant to detect fraud and when it did it was quickly adjusted and all the evidence was removed.

[OnTheOtherHand]

21. OK, that's fair

Sorry, I'm probably a bit gun-shy at this point! Let me back up and just try to answer your questions.

"Well then why did they have to change the numbers for that particular question when they delivered the final exit poll."

They don't pick and choose which questions they are going to change the numbers for; they reweight them all. They calculate a multiplier for every respondent that ends up matching the proportions of Bush and Kerry voters in the official count (as well as their estimates of the overall distribution of age, race, and sex among voters -- interviewers are supposed to track those variables for the people they aren't able to interview). Then they rerun every table using those weights.

Why do they do this? Because, assuming that the exit poll is off, using the weights should yield more accurate tables. (I sort of spelled this out, with a numerical example, in a post here: http://www.democraticunderground.com/discuss/duboard.php?az=show_mesg&forum=132&topic_id=2490768&mesg_id=2499695 .)

"Maybe that wasn't the point but when they re-weighed the poll to make it match the results, by increasing the number of people questioned who voted for *, they forgot about those numbers which look ridiculous."

I don't think they "forgot." I suppose they could have looked at the past-vote table and said, "Wow, some folks are going to be arguing from here to eternity that that table is evidence of fraud. Maybe we could, umm, jigger those numbers." But they didn't.

Now, here's the interesting thing I found out when I looked at a lot of old exit polls (and some other polls, too) -- generally more people say they voted for the previous winner than actually voted for him. In fact, I wasn't able to find an exit poll where the percentage wasn't higher than it "should" be (assuming that the people who voted for the previous winner, and loser, turn out at similar rates). Also, I looked at a panel study where people were asked in 2000 whom they had just voted for, and then asked again in 2004 whom they voted for. A bunch of people who said in 2000 that they had voted for Gore, said by 2004 that actually they had voted for Bush. I think it is mostly because a certain proportion of people, who don't follow politics very closely, pretty much just forget the loser. I wrote this all up here: http://inside.bard.edu/~lindeman/too-many.pdf

"Why did they even bother to make the final exit poll match the results the day after, to make it look like they had messed up less?"

Again, that's what that other post is about. Assuming that the exit poll is off, the reweighting ought to yield more accurate results for all the tables (although even then, it's possible for the weighting to backfire).

"If one believes that the election was fraudulent one must understand that the exit poll unwittingly would reflect that."

Yes, if there was massive vote miscount, the exit polls should reflect it. The exit polls may also reflect other error sources. Vote suppression (such as registration purges) probably wouldn't show up in the exit polls except in the form of uncounted provisional ballots (and by affecting the turnout estimates -- that's an arcane topic).

"The fraud is sometimes illustrated by precincts with turnouts in the high 90s, third party candidates getting more votes than they should, candidates from the presidential candidate's party obtaining more votes than them and possibly exit polls."

Huge topic. In Ohio, those first three things all happened, but not very often. Smart, decent people are going to go to their graves disagreeing about what happened in Ohio.

"I mean even if the methodology was slightly off it seems that exit polling is more accurate than actually voting."

They could both be way off. I think the exit poll was way off in Ohio because it put Kerry up 6.5 points, and that's just too many points IMHO, especially considering all the people who never got to vote. (We didn't realize that on election night because the initial tabulations -- the CNN.com screen shots -- are based on an estimate that uses both the exit poll interviews and pre-election expectations. Now we know that while that estimate put Kerry up 3.4 points in Ohio, the interview-only estimate had him up almost twice as much.) I also think the exit poll was way off in New York because (as in Ohio) the official result was a lot closer to pre-election polls than the exit poll result was. Not that I think pre-election polls are perfect by any means, but I sure don't think Kerry won New York state by 30 percentage points, either.

But the kicker is: even if the exit polls were wrong, it obviously doesn't prove that the count was right. (And even if the count was basically right, it doesn't prove that the election was fair, or that the machines can be trusted.)

I do think that the count was basically right (at least in most places). I get in a lot of trouble for saying it. But please reread the end of the previous paragraph.

Bear with me. It was a big election.

[BeFree]

25. Thanks, I see it!

But the kicker is: even if the exit polls were wrong, it obviously doesn't prove that the count was right.

(And even if the count was basically right, it doesn't prove that the election was fair, or that the machines can be trusted.)

Now we're talking.

Bad exit-polls can't prove the official counts were right. But good exit-polls could prove the official counts were bad.

So the question is: were the exit polls good, or bad?

Did machines steal votes? Yes.

We know the machines are built to steal votes and thereby make an election unfair. The official count was wrong because of the not to be trusted machines.

We know the estimates were modeled to be as accurate as possible. We can conclude that the estimates were good since they were not corrupted by the inclusion of what were bad official counts.

We know the final exit-polls were matched to the bad official counts.

So, since the final exit-poll numbers had the bad machine counts rolled into them - forcing the deviation from the good estimates - it means the final exit polls were bad.

Conclusion: The estimate parts of the exit polls, the ones claiming a Kerry victory were good, and the final, contradictory numbers were bad.

[OnTheOtherHand]

28. once more with feeling; the lever machines in New York

"Did machines steal votes? Yes."

You have evidence that the lever machines in New York stole votes?

"We know the machines are built to steal votes and thereby make an election unfair."

You have evidence that the lever machines in New York are built to steal votes?

"We can conclude that the estimates were good since they were not corrupted by the inclusion of what were bad official counts."

Even if we followed you in assuming that the official counts were wrong, this argument still would fail. You might as well argue that antifreeze must be good to drink because it doesn't contain cyanide.

OK, so, for the benefit of people who may not know why I am talking about New York: New York is an excellent case to examine because although it wasn't competitive, it had several late pre-election polls due to its size, and they were pretty consistent. In fact, four polls in the last week of the campaign showed Kerry up by 15, 16, 17, and 18 points (not in that order). Kerry officially won by 18.3 points. The exit poll estimate for New York (the one based on interviews alone, the Best Geo) showed Kerry winning by 31.3 points.

So if someone really believes that Kerry managed to outperform the pre-election polls by about 15 points, and then Bush managed to steal about 13 points back by hacking the lever machines, I would like to see some serious argument for any part of that belief -- or at least some evidence of serious interest in it.

Otherwise, it's pretty hard to conclude that "the estimates were good."

[BeFree]

31. New York again?

You just don't stop, do you? No one argues that the NY estimate didn't have some errors. Heck, I'll bet Mitofsky has lost sleep over NY. What does he say about NY? Email him and ask him.

Just points out why not even kool-aid drinkers would use estimates from just one state to decide a national election. Duh!

"One bad apple don't spoil the whole bunch, girl!"

[OnTheOtherHand]

32. I won't stop until you come up with an argument

If "(n)o one argues that the NY estimate didn't have some errors," then why does anyone argue that the exit polls are accurate?

More to the point, why do you argue it?

And this wasn't just "some error()," this was a double-digit error, bigger than the discrepancy in Florida or Ohio or Pennsylvania -- in fact, bigger than any state except... Delaware, and Vermont.

It's not a matter of using "estimates from just one state." It's a matter of people cherry-picking the exit poll discrepancies they like, and ignoring the ones they don't.

[bullimiami]

5. they started attacking the exit polling in 2000 and have

essentially destroyed it as an institution. the news orgs may get together again as they did in the last election and do their own exit polling but to what end?
in 04 they 'corrected' the exit polling to make it correlate closer to the count. and afaik the raw data was never released for examination.

so here we are in 08. the reliable pre polls and exit polls show a democratic landslide in senate, house and presidential races BUT the voting machines show the republicans winning everywhere, taking the presidency and even more house and senate seats.
I think this is a real possiblity because the more they have gotten bolder with every crime they have gotten away with.

What do we do? What will we do?

I dont think we can trust anything, reported results, polling, exit polling or news articles about it, without extremely critical consideration.

As an example, view the spin the MSM has put on the mexican elections. What is their stake in this that they feel they have to ignore obvious tampering, repeatedly demean the 'left' candidate, and essentially declare calderon the winner? They do the same to us. Why?

[passy]

7. Why ? Because dollar bills aren't red.

[karynnj]

9. The results of earlier years aren't used as weights for the exit polls

The Florida exit polls DID NOT match the "official" results. Likely because of the many errors that took votes cast intented for Gore away from him. The exit polls were for an unambiguous Gore win and were part of the reason that they called FL for Gore. (About 90% of the inner city Jacksonville vote was rejected as they voted Gore on page 1 and Brown on page 2 - thinking they were voting for the Congressman (the Democrats had a commercial saying to vote every page - not thinking there were so many Presidential lines that it went to page 2 on the punch ballots) and the well known Palm Beach problem.) Oddly, the exit poll was a more accurate estimate of who most people intended to vote for.

The 2000 experience led the TV NOT to declare based on exit polls. If you think about it, this hurt Kerry. Imagine that 2000 had not happened - and 2004 was covered using the procedures from 2000. People would have seen a huge Kerry landslide developing as the polls closed in various states - look back at the exit polls, Kerry was winning in many states (Ohio, NM, etc) that were later called for Bush. Other than people on the internet and those of us with huge smiles at various Democratic headquarters on election day, no one seems to know this. Remember that the cable news than trumpeted it as a huge Bush win - so he had lots of political capital.)

[Awsi Dooger]

10. Even minus punch card errors, the Florida exit poll was too kind to Gore

The projected margin was 2%. That would have been roughly 120,000 votes.

I have 12 VHS tapes from that election cycle and have watched portions of them many times. Mary Matalin goes nuts on election night disputing the early call to Gore. She says she's told the exit poll projection is 2% toward Gore, which is not enough to call a pivotal state so early in the evening (agreed), and regardless she thinks the absentees and other factors will favor Bush.

In a perfect tally the actual margin would probably have been close to 1%, maybe 55,000 votes. That's minus over votes, butterfly confusion, or excessive under votes. Purged voters is another matter, as OTOH indicated. But there had to be a minor sampling error to push the estimate to 2%. The exit poll firm said after the race there had been demographic changes in Florida since '96, and their model had not accounted for that. Amazing they were apologizing for perhaps a 1 or 2% error in 2000, considering what would happen four years later.

I'm still dumbfounded anyone from the networks or VNS gave the go ahead to call that race for Gore. Florida 2000 should not have been designated for either candidate on election night.

[OnTheOtherHand]

11. just a few more points

karyn drilled the main one for purposes of this thread: "The results of earlier years aren't used as weights for the exit polls." I tried to say that twice, but I'm too sleepy. ;)

The odd thing about FL 2000 as an exit poll is that the mean Within Precinct Error (WPE) was under 1 point, but the model estimate was from hell. Apparently the precinct sample was off, the absentee assumptions were off, their reference-race algorithm picked the worst possible reference race -- just about a perfect storm. And -- in part because those particular things were off -- the precinct quick-counts they were getting reinforced their confidence in the call that they were about to blow. Note that (1) they waited for vote counts before making the call, and (2) if the within-precinct error had been higher, they probably wouldn't have blown the call, because they would have seen a larger margin of error in their estimate. (I haven't seen the internal numbers, so possibly I am hyping the paradox.)

The model error to Gore was quite substantial, a lot more than 2 points -- more like 7. And apparently the early quick counts seemed to support it. It was only a bit later, when they looked at a wider range of vote counts, that the sinking feeling set in. There is no way that they would have called Florida for Gore based on a 2-point lead in the exit polls. Which, by the way, is probably one reason why Mitofsky absolutely scoffs at Steve Freeman for using mean WPE to extrapolate supposed exit poll projections and claiming that the 2000 Florida exit poll was discrepant. If Freeman really believes that mean WPE is the gold standard of exit poll accuracy, then Florida 2000 was about as accurate as it gets.

Unfortunately, none of this has much to do with all the real stuff that went wrong in Florida, like the huge overvoting in Duval County (Jacksonville) that karyn mentioned. (I don't think it was "90% of the inner city Jacksonville vote," but I've never been to inner city Jacksonville, so I will reserve judgment. It was almost 22,000 votes countywide, certainly more than enough to flip the state.)

EDIT TO ADD: for the 7-point error in FL 2000 (actually 7.3, I think), see Mitofsky and Edelman, "Election night estimation," Journal of Official Statistics 18, no. 2: 165-79.

[Awsi Dooger]

12. If that's the case, what happened with relationship to Prior Estimate?

Terrific detail, OTOH. But no way in hell any pre-election state poll or poll consensus gave Florida 2000 to Gore by 7 points, or anything in that realm. It was known to be the vital tossup state and generally mirrored Bush's small lead in the national polls.

Please don't tell me Prior Estimate was a new revelation in 2004. Maybe they just stick it in their literature but pay little attention, or did not in 2000.

Frankly, I was frustrated as hell when reading in that January 2005 Mitofsky PDF last week that Prior Estimate and everything they do with it is apparently restricted to the specific year. No understanding of historical voting trends, which are very reliable on the presidential level. Very often pre-election polls are simply laughable. For example, in fall 2004 one of the main polling firms, probably Gallup, came out with releases in Florida and Ohio on the same day. They had Kerry leading Ohio by maybe 8 points and Bush significantly ahead in Florida, maybe 6. So they were trying to tell us there was a 14 point gap between partisanship level in Ohio and Florida. Utter ignorant rubbish. We had a laughfest thread on GD 2004.

Those states are within a couple of points in partisan index and that's how they voted in 2004. Someone in that polling firm should have pointed that out and questioned the idiotic state polls. In that instance, of course, Ohio was the bizarre number and that was apparent at the time. Likewise, on election night 2000 I wish someone at VNS was evaluating returns and exit polls in other states and questioning how Gore could be "leading" Florida by 7 points when that didn't align in relationship to other states.

Wow, now I'll have to look at Florida 2000 in regard to exit polling. I always ignored that aspect and accepted Mary Matalin's claim of a 2% gap.

You know, I really wish I had that 7.3% ammo in November/December 2004.

BTW, karyn's point on massive number of center city Jacksonville over votes is good, but I think there is a misunderstainding of that 90% figure. As I remember it, those 22,000 votes were lost in precincts that voted 90+% for Kerry, not 90% of all attempted votes were lost.

[OnTheOtherHand]

14. well, that's a really good question

Even in 2004, they seem to have used the Best Geo (which doesn't incorporate the Prior) than on the Composite (which does) -- at least for the purpose of determining whether their formal significance criterion was satisfied. But it almost doesn't matter what estimates, specifically, they were staring at on election night 2000: no matter what they literally were looking at, you've got to wonder, when the results were so far from what it seems the priors must have been (or, to put it differently, so weird on their face), why did they not wait longer?

I guess the vote count data must have looked really compelling. I can imagine it. Have you ever had a 30-minute period where everything seemed to be falling into place, and later on you wondered, "What was I thinking?"?

By the way, I used to think that they used the Composite for their call decisions, because (1) it seemed reasonable, and (2) I misunderstood a sentence in the evaluation report.

Don't assume that the priors are based only or mechanically on pre-election state polls. I think they exercise some judgment (of course, I won't venture whether it is good judgment). Yeah, the Gallup poll for Ohio in 2004 was a wild outlier.

"Likewise, on election night 2000 I wish someone at VNS was evaluating returns and exit polls in other states and questioning how Gore could be 'leading' Florida by 7 points when that didn't align in relationship to other states."

Yeah. FL was one of the states that closed at 7 (except for the panhandle), so that may have compounded the confusion. --Oh, I had forgotten this: their early quick counts actually suggested that Gore had done even better than the exit polls said. I.e., their initial counts indicated blue shift.

Mary Matalin might've had some basis for that number -- lots of numbers fly around. Not that I regard Mary Matalin as a reliable source. (BTW there turns out to have been some interesting coverage of the 2004 exits on Fox, but I didn't know about it until last month or so -- I had never plowed through the transcripts.)

Ah, I couldn't figure out where karyn's 90% came from, but I bet you're right. I assure you that not all those votes came from 90+% Kerry, but the skew was large. I will be looking it up again at some point.

[Awsi Dooger]

20. Why doesn't Mitofsky also do pre-election polling?

That's another question I've had but been hesitant to ask. Just seems so bizarre a firm shows up on election day alone. At the same time the Rasmussens and Gallups and Mason Dixons, etc. do the workouts then sit out the race.

Obviously you're dealing with a tiny number of people in pre-election polling, and most of it by phone, so it wouldn't impact lack of training problems among the exit pollsters. But if Mitofsky wants to gauge how accurately his exit poll results are, wouldn't he be more comfortable if they lined up with his own Prior Estimates? He could even design and prioritize questions in that regard, an understanding of what percentage in each state answer specific questions a certain way and then see if that is similarly reflected in his exit poll sample. I don't think it's such an impossible transition; how will you vote as opposed to who did you vote for?

I guess the pre-election polls, if released, would lead to inevitable claims of bias one way or another depending on how they compared to other polls, or conventional wisdom about the race. Right now one could argue exit polls themselves are stereotyped as tilting left. Based on what have you done for me lately.

Nothing about Florida 2000 makes an sense. Other than it figured to be very tight and turned out that way. I was watching just before 4 PM (7 Eastern) here on the West Coast when they announced major calls coming at the top of the hour. I said to myself, it can't be Florida. When it was, I shouted for joy then wondered how that was possible, unless Gore was headed for a 5 point win nationwide.

BTW, I was taping CNN, not FOX, and that's where Mary Matalin later appeared, tortured look on her face, even more than normal. Maybe I'll find the tape and watch it again, but her words were basically, "I've been talking to our people who have seen the numbers and the model is two points for Gore."

Uh, yeah, living in Las Vegas and frequenting sportsbooks I've had many 30-minute periods like that; what type of new car turning into, do I have anything in the fridge? :)

[OnTheOtherHand]

23. I think Mitofsky is very happy with his niche

Now, when you look at the Edison/Mitofsky partnership, apparently Mitofsky specializes in putting the precinct sample together and working on the estimation models, and Edison basically conducts the polls. Edison does do lots of other kinds of work. As for Mitofsky, I don't think he lacks for work these days -- he isn't kidding about the "International" part. No one is asking him to start running new, improved state polls at their expense.

"But if Mitofsky wants to gauge how accurately his exit poll results are, wouldn't he be more comfortable if they lined up with his own Prior Estimates?"

I wouldn't put it that way, because of course he has prior estimates. Does Mitofsky think that he could do pre-election polling much better than the people who do it now? I don't know -- but the several thousand interviews per state he would probably need in order to know for sure isn't a cost that one would eat just for the sake of being 'comfortable.' And how could he actually know that his poll was the accurate one, anyway? Bear in mind that the pre-election poll problem is inherently intractable: even the respondents don't know for sure whether they will vote, or whether they will change their minds, so how can you? Even if you get the "right" answer (and even if you could be assured that the count was accurate, so you really did get the right answer), how would you know how many design decisions you got right, and to what extent you just got lucky?

That's why so many people decided that exit polls were cinches by comparison. Oh well.

As you know, the existing pre-election polls are already political footballs. It's pretty interesting.

[Awsi Dooger]

26. I wondered if Edison and Mitofsky had different roles

Very interesting. So where were the bulk of the flaws in '04, the precinct sample/estimation models, or the polling itself? You mentioned exit poll model from hell regarding Florida 2000. If every state had WPE near zero in 2004, would any state have had a significantly flawed model?

My question stemmed from the January 2005 PDF and the need for "further experimentation." Additional polling of any type in this country would seem to be beneficial in that regard. And it's certainly much easier for an exit pollster to incorporate pre-election polling than the other way around, similar to a tennis player naturally excelling at ping pong while the reverse is a major leap.

Well, I just popped in the VHS tape from November 7, 2000. I was planning to fast forward to Mary Matalin's comment on the 2% gap but now I'm fascinated enough to watch all the way through. It cracked me up within the first half hour of coverage when CNN cut away to a reporter in Rancho Mirage for an update on a new technology, electronic voting machines. Very upbeat report, interviewing several senior citizen types who marvelled at how easy the machines were to use, "especially if you come prepared." No mention of paper trails or thievery. I couldn't help thinking two things; how different a report like that would be now, and how much I wouldn't have given to transport those machines to Dade, Broward, Palm Beach and Duval.

[OnTheOtherHand]

29. well...

In Florida 2000 (just for purposes of comparison), the precinct sample was off. In 2004, everyone agrees that the main problem was with the interview data, not primarily the precinct sample. That doesn't necessarily mean that the polling firm did something wrong -- or at least, it doesn't mean that their errors, whatever they were, caused the problem. Complex systems aren't about just deserts. Also, it isn't just a talking point that in fact, E/M didn't blow any calls in 2004. I read the CNN transcripts last night, and the coverage (albeit fluffy) was just fine on key themes. The whole idea that 'everyone thought they knew from the exit polls that Kerry would win' (not that you said that -- I'm rambling) seems to play very well here in political cyberspace, but I don't know about the rest of the world. So the pollsters and the networks aren't going to treat this like the Challenger disaster, because it wasn't. (Of course, they may not learn as much from it as they should. We will see.)

"If every state had WPE near zero in 2004, would any state have had a significantly flawed model?" Stare at pp. 29-30 in Evaluation of Samples. "What's the matter with Kansas?" From a straight-up stats point of view, that might be more interesting than most of the exit poll debate, but let's face it, no one cares about Kansas. (Seriously, I hope and suspect that the pollsters have taken a good long look at Kansas, just in case it points to a flaw that could bite them in a swing state next time around.)

Sure, more pre-election polling might be revealing; I just don't see anyone paying for it. (And FWIW if I were known as "the father of exit polling," I would tend to devote my energy to experimenting with those methods, not the others.) I'm not sure that going from exit polls to pre-election polls is much like going from tennis to ping pong -- but then, I never played tennis well enough to go to ping pong from it. But there is a lot of 'dark art' either way, and it is different dark art (setting aside the sheer physical logistics of the exit polls).

[Awsi Dooger]

33. Well, I knew the answer to my primary question

That it was the data, not the models in '04. But I've picked up so much in this little thread it's worthwhile to pester OTOH for knowledge, like the states deviating from model. That is fascinating about Kansas. I never noticed that before. Also Mississippi, to a lesser degree. What do Freeman and others say about Kansas? Probably not much. I just Googled and came up with zilch regarding that situation.

Yeah, I was referencing physical differences in the weak tennis/ping pong analogy. Much simpler for an exit poller to put together a phone setup for pre-election polling, than for pre-election pollers to form a ground crew in state after state and city after city, being faced with all the new and unknown variables, like where they're allowed to be. Tennis players asked to stand behind a ping pong table is similar to phone banking, while ping pong players forced to run and cover a much larger territory than they're accustomed to is somewhat akin to pre-election pollers trying to exit poll. That's what I was going for, not implying exit pollers would be superior at pre-election polling. Oh, never mind.

Regarding the end of your first paragraph, what if election 2004 had been held in 1992, or any year prior to the impact of the internet? No blown calls. Probably no hysteria from professors in unrelated fields, or rage against the machine. Minor TV and print chatter about exit polls being unusually off. Actually, maybe not. Remember, in '92 the exit polls were a much smaller undertaking. :)

For anyone embracing the exit poll process as sophisticated and infallible, I really wish they would read that January 2005 PDF. Here's one paragraph I found amusing: "There were 62 polling locations in which a replacement interviewer was sent on election day to replace an interviewer who did not show up or to replace an interviewer who had to leave for some reason. Since some of the replacements arrived late, some exit poll calls were not made from some of these polling locations."

[OnTheOtherHand]

34. "bother bother bother bother bother" ;)

Dunno if you are acquainted with Potter Puppet Pals -- our daughters would just Love to bring you up to speed, if not.

I don't fault Freeman for not writing about Kansas -- even I haven't been that hard-core -- but it's worth noting that the preferred narrative has been "everyone agrees that the precinct sample was accurate, so it's down to either rBr (pause for hissing from the audience) or KERRY WON! (jubilant huzzahs)." Overall the precinct sample was accurate; I'm snorting a bit, but not really objecting. Freeman does tend to make things sound a lot more slam-dunk than they are.

OK, it's a fair cop: the physical logistics of phone polling (while more complicated than some folks might think) clearly don't begin to compare with the logistics of 1500 interviewers calling in to a phone bank. And there is a phone component via the absentees anyway. Still, the unit cost is pretty steep.

I think you are exactly right: if 2004 had happened in 1992, probably almost no one would have noticed. There was some minor chatter in 1988, because that year there were actually two blown calls (I think one by ABC, one by CBS). Which just goes to underscore that blown calls don't correlate with overall bias. No blown calls in 1992 or 2004.

"There were 62 polling locations in which a replacement interviewer was sent on election day to replace an interviewer who did not show up or to replace an interviewer who had to leave for some reason. Since some of the replacements arrived late, some exit poll calls were not made from some of these polling locations." What's not so amusing is that there is a pretty similar paragraph in the Franklin County BoE's report on its deployment of voting machines. According to the BoE, it tried its damnedest to deploy practically its entire reserve. There's actually a document that seems to imply otherwise, and I don't know whether the BoE has an answer to that. But I didn't find their story inherently far-fetched.

[Awsi Dooger]

35. This might be what you're referring to

http://centralohiovoters.home.att.net/elect2004.htm

"In the past, voter registration growth and anticipated Election Day turnout did not warrant the widespread assignment of these reserve machines. However, Board staff wisely determined that the Election environment in 2004 did indeed necessitate the use of these machines. To that end, seventy-seven of the ninety-nine reserve machines were assigned for delivery and use following the safe delivery of the final load of machines by the contract hauler on November 1. However, only sixty <59 according to Table 3> were actually delivered and used.

There was an internal breakdown in communication on the total number of reserve machines that were to have been assigned on November 1 resulting in only forty-four machines being assigned the day before the Election. There was a subsequent breakdown in communication on Election Day that resulted in twenty-two voting machines never being assigned for use at all. Finally, the lengthy time necessary to assign a machine, print its ballot, program its machine cartridge, perform the set-up logic and accuracy tests, and then deliver a machine -- and personnel constraints for performing these and other Election Day tasks -- reduced the ability of the Board to nimbly and timely respond to significant increased voter turnout."

Well, I'm not sure if that's a net positive, wise but lacking nimble and timely. Actually, I did find it amusing, self congratulatory on the front end then listing every conceivable problem on the way out. What happened to, "no time to blow off the dust."

Anyway, wasn't there a blown call in Florida in 1992 also? I saw GW say that once in an interview, that Florida was called for Clinton but his father, Bush 41, ended up winnng the state. I immediately looked it up and it appeared to make no sense, since Bush Sr. won the state by roughly 100,000 votes and nearly 2 points. However, that was the big Perot year so a 3-way would be more difficult to exit poll, and potentially more gaffes. Maybe it was only one network or one media organization that called the state wrong but I'm fairly sure I saw GW say that.

Also, I've been thinking about this since you mentioned the flawed exit polls in '92. The oft-quoted numerical hoopla about the exit poll discrepancy in '04 as such a once in forever and beyond event. But what about 1992? If those exit polls were significantly off, I'm sure you could look at the individual states and calculate a theoretical percentage there also, something that would look ridiculous if you lined the '92 and '04 numbers up together and tried to claim it was real world. Now, it would be dramatically less not fractionally less, something akin to most sophisticated DNA pattern as opposed to much more crude and limited, but still stupid to quote, like saying the Patriots were 9 billion to 1 in '01, 50,000 to 1 in '02, then only 1000 to 1 in '04.

Nope, not familiar with PPP.:)

[OnTheOtherHand]

39. yeah

In fairness to the Franklin County BoE -- well, I don't know for sure what is fair to the BoE, but they were accused of sabotaging the election by deliberately holding machines out, so it's not surprising that they were defensive. As for the prior machine allocations, the most charitable interpretation is that they made them based on stale data, and when the new registration data came in, they just weren't willing to accept the responsibility to take machines out of any precinct to put them into another precinct that was even worse off, so they just crossed their fingers that Blackwell would approve some accommodation (it's not clear what accommodation they were hoping for -- something to do with paper ballots). Apparently Blackwell, characteristically, said, 'But we couldn't let people VOTE! why, that would be tampering with the DEMOCRATIC PROCESS!', and that was that. I hope to get a bit closer to the bottom of that story before I'm done.

"Anyway, wasn't there a blown call in Florida in 1992 also?"

Not that I remember, but there is much I have forgotten or never knew -- and every network does have discretion to blow its own calls. I think Clinton would have been slightly ahead in the 1992 Florida exit, and we know from _The War Room_ that he was one point up mid-day (I don't know what the Call 3 estimate was). No one would even think about calling a state based on a one-point lead in an exit poll (even if it got a bit bigger by the end of the day). So I'm skeptical.

"The oft-quoted numerical hoopla about the exit poll discrepancy in '04 as such a once in forever and beyond event. But what about 1992?"

Febble made your point in a memorable April '05 Daily Kos diary titled "BREAKING: One in Nine Billion probability that Exit polls could have happened by chance!" Characteristically, she was then a bit sheepish about having been so snarky.

I am going on the road for the rest of the week, so

[Awsi Dooger]

44. Looks like I never think of anything first

OTOH, thanks for giving me the exact title of Febble's post. That enabled me to Google it and it popped up first. Remarkable diary, including the comments.

Her number is 1 in 5007 in 1992. Not only that, but the other years amazed me also, 1 in 49,827 in '98 and 1 in 268 in '96. All overstating the Democratic vote. I must have seen her 9 billion to 1 odds quoted somewhere because that's a number I remember hearing in regard to '04, but it could have come from Freeman or TIA also.

http://www.dailykos.com/story/2005/4/2/174011/1181

[Awsi Dooger]

27. Mary Matalin's comment on election night, after Florida called for Gore

More focus on absentee votes than I remembered. She said she had been looking at some numbers and the "spread" was 2%, but she was convinced that would be made up via absentee votes, supposedly 500,000 of them. Earlier, before Florida was called, she said the Bush camp was expecting a +150,000 net via absentee votes. That was obviously the talking point from the right since Robert Novak said the same thing on a brief election night version of Capital Gang.

In watching the tape, it's unclear if she is referring to the percentage difference of the actual vote at that time, very early in the night and before CNN had shown any actual vote totals from Florida, or the 2% is the projected exit poll spread, which is what I always believed.

[karynnj]

13. It could well be an error in the sample design

The sample is designed more to study the voting patterns than to predict the total. For this reason, the precincts to be sampled are not randomly chosen. This creates the possibility that they differed from those not chosen in some systematic way that was not based on the variables they used to expand to the total population.

[OnTheOtherHand]

15. it depends on what one means by "random"

I don't have any first-hand knowledge of this, but I've read the reports and asked Mitofsky some follow-up questions via e-mail. They don't hand-pick the precincts, nor do they do a Simple Random Sample; they use a random method that is designed to ensure that the partisanship spectrum is evenly represented.

But you could be hinting at something else; I'm just not sure what. It's 86 degrees here in my study....

[passy]

18. Excuse me, could you explain this.

"they use a random method that is designed to ensure that the partisanship spectrum is evenly represented"
How do they evenly represent the the partisanship spectrum? What method can they use to do that accurately?
Do they base it on how people voted in previous elections or on how they have registered?
Did they adjust their numbers to match the difference in new registrations among democrats and republicans?
I thought that democrats had a much larger number of new registrations than republicans, but that didn't seem to be reflected that much in the final results. I guess many of these newly registered democrats ended up voting for *.

[karynnj]

22. I assume that they use some form of a 2 stage stratified random sample

They likely place the precincts into clusters based on pertinent demographic characteristics, then randomly sample precincts from each cluster. My comment was not to say that their technique was bad- clearly very skilled statisticians would be used and this would be a sensible method.

If the process divides the precincts into groups that are pretty homogeneous in terms of the percent of vote for each candidate (similar to each other) and the groups differ from each other, this could lead to an estimate that is more precise than you could get from a simple 2 stage sample where you simply sample the precincts then randomly sample voters within the precinct. If this is not the case, you still get a valid estimate but with a wider than expected confidence interval - so you could be say 2% off in either direction.

This was the type of explanation I had expected in 2004 - not the shy Bush voter - which never happened before and given that the government, media and many churches urged people to vote Bush, the shyness is inexplicable. Given the church involvement, I would buy a shy Kerry voter argument in traditional conservative Catholic areas. To me it seemed they were saying, the design was ok, the analysis accurate - and the projection differed from the "official" result - ASSUMING THE OFFICIAL RESULT IS "TRUTH" - the implementation of the sampling of the voters had to be bad, thus the "shy voter" phenomenon.

[OnTheOtherHand]

24. geostrata, mostly

It's fun to 'talk' with someone who knows what a sample is.

From memory (which is risky for me after several consecutive days over 90 degrees), they stratify the sample into 'large' and 'small' precincts (I don't remember the deets, but the ultimate designed is supposed to give each voter roughly equal p of selection) and geostrata (typically 'regions' of states, but I think not always contiguous). Within each stratum, they sort the precincts by partisanship, use a random starting point, and 'drill down' to obtain the appropriate number of precincts with a balanced partisanship distribution (again, each voter has equal p of selection). That's the flavor of it, anyway.

Actually, the evaluation report shows that the last five presidential exit polls all had significantly negative mean Within Precinct Error (5 points in 1992, about 2 points in 88/96/00). Not necessarily all of that would bias the models -- for instance, there are circs where absentees could lead to awful WPEs but good model estimates. Model estimates aren't available for all those, but we know that there was considerable bias in the estimates in 1992, and the WPE was large then. I think I looked at ABC 1984 and that appeared to be biased too (unless I missed a weight I was really supposed to apply -- most of the ABC exit polls were designed to be self-weighting, whereas the CBS/Mitofsky ones really aren't). The 1980 ABC poll seemed pretty accurate. Overall, not really all that pristine a record of freedom from bias. But usually good enough to avoid bad calls (and actually, there were no wrong calls in 2004).

I think the exit pollsters tended to downgrade the likelihood that the polls evinced massive fraud for several reasons: the fact that they had seen poll bias before; the fact that many of the exit poll estimates didn't jibe with their priors (they actually held a conference call with the sponsors that afternoon to warn about some of the figures) ; the fact that WPE didn't vary by machine type in ways predicted by fraud hypotheses (although this point is contested, at least at the margins) ; the fact WPE did covary with some other things, such as interviewer distance from the polling place (which doesn't make much sense unless there is some underlying bias to be exacerbated by distance -- although that point, too, has been contested). I don't think they just assumed that the official count was right, although they certainly didn't assume that the poll was right.

There's also the interesting fact that red shift isn't correlated with change from 2000 to 2004. It seems really hard to believe that, say, 20-point red shift was targeted to precincts where Bush needed those 20 points just to break even. I didn't do the work on this, I just host the slides: http://inside.bard.edu/~lindeman/slides.html

I agree with the exit pollsters: a clear preponderance of the evidence indicates that the polls were off. And some of the results are hard to square with Bush having won the popular vote (see the link above). The exit poll data are just too sparse and noisy to shed much light on what happened in individual states (I do think they have enough power that they could register large fraud, but they certainly don't have enough power to infirm relatively small fraud -- and they don't really measure suppression).

[karynnj]

30. Thanks for the detailed reply

You really looked into this far more than I did. Thanks for the link, I'll definately look at it.

[eomer]

36. How do they determine partisanship of a precinct?

passy said upthread:

"they use a random method that is designed to ensure that the partisanship spectrum is evenly represented"
How do they evenly represent the the partisanship spectrum? What method can they use to do that accurately?
Do they base it on how people voted in previous elections or on how they have registered?
Did they adjust their numbers to match the difference in new registrations among democrats and republicans?
I thought that democrats had a much larger number of new registrations than republicans, but that didn't seem to be reflected that much in the final results. I guess many of these newly registered democrats ended up voting for *.

In your step where they "sort the precincts by partisanship", what metric do they use as an indication of the partisanship of a precinct? Is it official vote count from prior elections, voter registration, some other data, or a combination?

[OnTheOtherHand]

37. I'm not sure

I assume they use vote count for a reference race or races. They wouldn't use voter registration because in many states that would be quite unedifying. Bear in mind that their models look at deviations from the reference race or races. I'm not sure why that matters without knowing exactly what you're doing, but I bet it matters somehow. At any rate, they don't depend upon each sample exactly matching the mean partisanship of the geography (if "mean partisanship" even means anything). They are basically trying (in my understanding) to make sure that they span the breadth, so they are less likely to miss a shift in some part of the spectrum just through the mischance of not having sampled there.

[eomer]

38. But can you clarify whether this amounts to weighting or is it something

else?

Maybe we can assume for now that partisanship is based on official vote count for a reference race or races in one or more prior elections.

Once they've got a sample that spans the breadth, does the partisanship proportion of precincts in that sample constitute weighting that will determine (partially) the exit poll output? Or is the sample up to this point still just a bunch of candidate data points to which weighting will be applied later in the process so that the weight in the original sample by partisanship is thrown out and replaced by a weight determined some other way?

BTW, what I'm doing is coming back to the original question in the thread, which was whether historical errors could affect future exit poll predictions. This is more or less the same question I've been picking around for the last few weeks. If the official count is incorrect and then is used to develop weighting for the exit poll the next time around, what effect will that have?

[passy]

40. I think you get my point eomer.

If you look there, someone has done a simple calculation of potential errors that could arise: http://www.progressiveindependent.com/dc/dcboard.php?az=show_mesg&forum=120&topic_id=3034&mesg_id=3034

[eomer]

43. Thanks for the link.

I read it quickly and then realized I will have to go over it more slowly (TIA often has that effect on me).

I've been thinking along the same lines that you are and will keep looking into it. I'm afraid though that we will reach a point where we need to know the specific model that they use in order to answer your question. That's the point at which OTOH and I left off last time we had a similar discussion. We won't be able to know the details of the model since they don't publish it. Since OTOH said he is willing to work with me, I will probably develop my own model, or maybe a couple of variations that can explore which kinds of models would be affected by prior error in the official count and how they would be affected.

[OnTheOtherHand]

41. whether I can "clarify" remains to be seen ;)

I would say that it isn't weighting, it's just one way of 'spreading around' the sample precincts relatively evenly. As I said, the models look at deviations from reference races, so having e.g. 'too many' Republican precincts (I'm not trying to describe a fraud scenario) doesn't necessarily affect the results. (Crudely, if a Dem is running two points ahead of the reference-race Dem across the board, the model should put him two points ahead of the reference-race Dem, regardless of the precinct mix in the sample.) That said, how the models (there are several distinct estimation procedures) might interact with various fraud scenarios -- and what partisanship might have to do with it -- I certainly wouldn't venture an opinion any time before noon. You would need to consider QA filtering, too. And we don't know very much about any of it.

I think your best bet might be to build yourself a little simulated state and state exit poll (but maybe with more power, so you can see what it is doing) and see how the poll responds to various historical fraud scenarios. I can at least help you sort out whether your poll modeling design might bear some family resemblance to something the pollsters might conceivably do. There are a couple of articles that indicate some of the basics.

That said, I will be offline from Soon until probably Monday morning.

[eomer]

42. Well, I should probably be getting some real work done anyway...

so I'll resist throwing a couple more punches after the bell has rung.

Have a good break (if that's what you'll be doing). The modeling you suggest sounds interesting - I'd like to talk further when you are back.