There Is More To Poll Results Than Meets The Eye
In a previous post an average of polls was cited as evidence for Senator McCain’s likely win over Senator Clinton in the general election. The problem with averaging polls is that it mashes really excellent results with truly horrid ones, resulting in some mediocre mess as well as averaging extreme with more typical results. would like to argue that instead of averaging polls a more nuanced approach would be to find the best poll(s) and use that. So let us look more closely at those survey data.
First the “average” was composed of four values, all showing Senator McCain in the lead by 2, 3, 4, and 11 points. Now doesn’t that seem to say something difference than an average of 4? Let me continue. The undecided vote reported in each of those polls is 0, 1, 7, and 13, in that order. So the poll showing McCain with an 11 point advantage has 13% undecided while the one showing McCain leading by 2% shows zero undecided. You don’t have to know a lot about statistics to notice that these things are very different. So what happens when you “average” them? Is that outcome more precise? A better approximation? I think not.
So who do we believe? If three are close together and one very different, that would be a good start. But Surveyusa recently did a metanalysis examining the results from twenty four firms representing all of the polls done for all of the primaries and caucuses in 2008. (Yes a firm that does polls, but they show us the data.) They combined all of the polls done by firm and calculated the average error, then presented the results in order of combined. An error of zero would indicate perfect prediction.
The best “average error” was 1; the worst was 10. Examining the firms whose predictions were averaged in the first presentation we find that two had an average error of 9, one had 7.7 and one was not available. Fourteen firms had a better result that the best of these. Beyond that a simple scroll down the page will show that Rasmussen, for instance, had consistently greater error when polling democrats (12, 10) than when polling republicans (8, 5, 8, 3).
I am not going to go through every detail because, if you have gotten this far it probably took two expressos and a piece of pie. I could have looked at how much change in the data we see over the past three months and point out that the election isn’t for ten more months, but I think you get that. My point isn’t that I know the answer. My point is that none of us do, including the “experts.” And that squidgy data like these are a lousy basis for decision-making.
Posted: January 22nd, 2008 under Engineering, Oversight, We Are the Enemy.
Comments: 4
Comments
Comment from Sasha
Time: January 22, 2008, 7:23 pm
Yes I am a total geek. Why do you ask?
Comment from Josh Hammond
Time: January 23, 2008, 7:50 am
Surprise, surprise, the firm that did the meta-analysis came out the best!?
The reason I cited realclearpolitics.com is because they do two things that no other site does, as far as I know: 1) they show all the polls they include and then they average them for what it is worth, nothing more than that–a research shortcut; 2) they show the trend over time that I find very interesting and helpful. I am not citing the average as “proof” of anything, except as a convenient way to start a conversation. I also don’t want to see McCain as president, in case, some think that is why I brought McCain into the storyline.
As I have stated before, I have worked with pollsters over the years, and in my work, I have found them to be incredibly accurate, given the margin of error and other conditions they use in every case, including a statement about their level of confidence in the findings. So I find it easy to defend the top level pollsters, or survey firms as they are more accurately known.
Not all polling firms are equal.
The Des Moines Register poll has historically been incredibly accurate on the Iowa caucuses. They were exactingly precise this time around, picking Obama to win: there was no “Bradley Effect” in Iowa in their poll.
Those who “hate” polls can easily point to one big screw up: the missed call in New Hampshire on the Democratic side, the same firms who got the Republican side totally correct. Now it seems that the burden is on the poll-dumpers to point to another screw up by a major polling firm. Remember that Clinton had a double digit lead in New Hampshire two weeks before the primary, and Clinton always had the lead in Nevada.
If you check realclearpolitics.com and look at the South Carolina polls, you will see that McCain closed the gap in the final few weeks, and he won by about the same margin as predicted, this in a state that Huckabee was suppose to win.
Bottomline for me: polls are like a thermostat that likely voters (the most reliable sample) set. What the acutal temperature is in the voting both is often, but not always up or down a degree or two. Less than 10 percent of voters make up their mind at the last minute, accounting for this kind of variance.
Comment from Sasha
Time: January 23, 2008, 5:43 pm
Mr. Hammond let me assure you that I chose my examples and sites carefully. And I think my years teaching statistics and research methods qualify me to speak on the topic. I don’t think I actually need to post the rest of my resume.
Comment from test
Time: April 1, 2008, 1:50 pm
test
Write a comment