Making sense out of Gallup and other presidential polls

The pollsters at Gallup have been on the defensive after issuing tracking polls showing Mitt Romney with a significant lead over President Barack Obama. But is the poll’s own track record that much different than other polls?

Frank Newport, the head pollster at Gallup, told Fox News in an interview over the weekend that the poll was “extremely solid.”

“It’s not unusual, going back to Dr. George Gallup who founded our company… He found heated commentary from either side on polls, and I’ve certainly found it in the six election cycles going back to 1992,” Newport told Fox. “People come at you from either side if they don’t like the results.”

Gallup also had a detailed explanation of its polling definitions on its website.

Newport’s comments came after a drubbing in the media about the “likely voters” section of the poll, which shows Romney with a 7-point lead over Obama.

Nate Silver, the New York Times political blogger, laid out a detailed argument against the poll in a post called “Gallup vs. the World.”

“We tend to put too much emphasis on the newest, most widely reported and most dramatic pieces of data—more than is usually warranted,” Silver says.

“Apart from Gallup’s final poll not having been especially accurate in recent years, it has often been a wild ride to get there. Their polls, for whatever reason, have often found implausibly large swings in the race,” he adds.

The website The Hill was less generous to Gallup than The New York Times. Allan Lichtman, a professor at American University, challenged Gallup’s concept of likely voters.

“You don’t have to compare the Gallup poll with any other survey to expose its flaws. A check of the internal consistency of this poll yields such implausible results that its findings are almost certain to be wrong. The key to this analysis is comparing results for registered voters and for likely voters,” Lichtman wrote on The Hill.

He shares Silver’s concerns about publicity related to the Gallup poll.

“Poll-driven journalism is not just meaningless. It is also pernicious because it detracts from coverage of what this election means for the American people,” Lichtman concluded in his story.

Constitution Daily did a little digging on Gallup’s own website to look at the track record of the tracking poll’s predictions for mid-October, going back to 1976, and the actual popular vote on Election Day.

We looked at 14 examples where Gallup’s likely-voter polls and registered-voter polls were taken closest to October 20, including five likely-voter polls and nine registered-voter polls.

Overall, Gallup had the correct winner in eight of nine presidential elections. (It had Jimmy Carter ahead in October 1980.)

Recent Constitution Daily Stories

Mitt Romney takes lead in projected electoral vote count
Inside America’s first dirty presidential campaign, 1796 style
Romney’s ace could come from a mystery swing state

Where Gallup diverged was in the difference between the projected national direct voting and the final November outcome.

The average Gallup October 20 popular vote prediction was at least 6 percentage points off the final election outcome. For example, Gallup had George W. Bush ahead by 6 percent over Al Gore among likely voters in October 2000. Gore took the popular by 0.5 percent, so the Gallup poll in October differed by 6.5 percent.

The closest Gallup came to calling the final results was in 2004, when its registered-voter poll had Bush ahead of John Kerry by 3 percent in October 2004. The final difference was 2.5 percent in the popular vote.

The average Gallup difference from the final vote was 6.2 percent in likely-voter polls, 5.8 percent in registered-voter polls, and 5.9 percent in both polls.

To be sure, Gallup is measuring voter sentiment about three weeks before an election, and a lot can change in that period. For example, Ronald Reagan’s dramatic debate victory in 1980 made that race into a rout.

And what about other polls? Are we just piling on Gallup just because they show a gap in the race between Romney and Obama?

Only five of 10 major polls taken in mid-October 2008 had the Obama-John McCain race within 2 points of its final outcome in November. Gallup’s traditional poll was off by 4.3 percent, while Pew was off by a whopping 6.7 percent.

Back in 2004, only three of 10 polls we looked at for a period that ended closest to October 19 were within 2 points of the final election result, and another two were within 2.5 points of the final results.

So out of those 20 polls in the past two elections, taken in mid-October, only about half were close to indicating the final margins on Election Night.

In mid-October 2008, Gallup’s expanded poll (which didn’t factor in past voting behavior) was very close to the final outcome of the Obama-McCain race, giving Obama a 7-point lead (he won by 7.3 points).

The uncertainty over poll outcomes in the general election has led to various folks who do “composite” polls that average and weight groups of polls based on past data.

The popular website Real Clear Politics keeps historical data about its accuracy in past elections. On October 22, 2008, the site had Obama ahead by 7 percent in its composite poll—which was very close to the final election outcome.

Today, it has Romney ahead by 0.8 percent in the popular vote.

Gallup Poll Mid-October Trends, 1976-2012 (Source: Gallup.com)