Daily Archives: June 4, 2017

Are The U.K. Polls Skewed?

In April, when U.K. Prime Minister Theresa May called for a “snap” general election for June 8, polls showed her Conservatives with an average lead of 17 percentage points over Labour. Such a margin would translate to a giant majority for Conservatives: perhaps as many as 400 of the 650 seats in Parliament. (Conservatives currently control 330 seats; 326 are needed for a majority.) After several unpredictable years in U.K. politics — marked by Conservatives unexpectedly winning a majority in the 2015 general election, the successful Brexit referendum, and David Cameron’s decision to resign as prime minister and Conservative leader — such a result promised to provide May with a mandate as she negotiated the terms of the U.K.’s exit from the EU.

Instead, polls suggest that the Conservative majority is under threat. As we remarked back in April, May’s move was riskier than it seemed because polls in the U.K. have been both highly volatile (shifting abruptly over the course of election campaigns) and fairly inaccurate (often missing the mark on Election Day itself). Conservatives’ lead was wide enough in April that they probably needed multiple things to go wrong to lose their majority. But if there there were both a shift toward Labour during the campaign and a pro-Labour polling error on Election Day, it could be at risk.

The first part of May’s nightmare scenario has come to fruition. Recent surveys show Labour zooming up in the polls and Conservatives having declined somewhat (although some of Labour’s gains have also come at the expense of Liberal Democrats and other parties). Conservatives’ lead varies from poll to poll — it’s as large as 12 percentage points in one poll and as little as 1 point in another — but it averages about 7 points in recent surveys, less than half of what it was when May called the election. The timing of the shift partly coincides with the release of the Conservative party manifesto two weeks ago, which included a proposed change to health care spending that opponents soon labeled as a “dementia tax.” (None of the polls yet reflect any potential effects from an incident on Saturday night at London Bridge, when a van reportedly hit a number of pedestrians.)

POLLSTER CON. LAB. UKIP LIB. DEM. OTHER LEAD
Survation 40 39 5 8 9 Con. +1
ICM 45 34 5 9 8 Con. +11
ORB 45 36 4 8 7 Con. +9
ComRes 47 35 4 8 5 Con. +12
Opinium 43 37 5 6 8 Con. +6
Ipsos MORI 45 40 2 7 6 Con. +5
Panelbase 44 36 5 7 8 Con. +8
YouGov 42 39 4 7 8 Con. +3
SurveyMonkey 44 38 4 6 7 Con. +6
Kantar Public 43 33 4 11 8 Con. +10
Average 43.8 36.7 4.2 7.7 7.4 Con. +7.1
Recent UK polls show a tighter race

Sources: UK Polling Report, Wikipedia

Conservatives won only a slim overall majority in 2015 despite winning the popular vote by 6.5 percentage points because 87 seats went to regional parties (especially the Scottish National Party, which won 56 seats), third parties (such as Liberal Democrats) or independent candidates. While FiveThirtyEight isn’t attempting to translate votes to seats — that’s a tricky problem and one that we haven’t had much luck with in the past — other people’s models show that if Conservatives were to win by much less than their 2015 margin, their majority would be under threat. A series of YouGov models released this week have shown Conservatives winning 308 to 317 seats — short of a 326-seat majority — with a 3- to 4-point win. Uniform swing calculations also suggest that Conservatives would be underdogs to retain their majority with a 4-point win, and about even money to do so with a 5-point win.

So to borrow our phrasing from the U.S. election, when we said that Donald Trump was only a “normal-sized polling error” away from winning the Electoral College, May’s Conservatives are now only a normal-sized polling error away from a hung parliament. On average in the U.K., the final polling average has missed the actual Conservative-Labour margin by about 4 percentage points. (This is twice the average error in U.S. presidential elections.) If Labour outperforms its polls by that margin, Conservatives would win the popular vote by only about 3points — and May would probably have to find a coalition partner to form the next government. If the polls were to miss by any more than that in Labour’s favor, a variety of yet-more-unpleasant scenarios could crop up for May, including some where Labour leader Jeremy Corbyn tried to form a government instead.


But there’s a catch — and a potential saving grace for May. Although the polls haven’t been very accurate in the U.K., the errors have usually run in the same direction: Conservatives tend to beat their polls there. (There’s been no comparable phenomenon in the U.S., where polls have erred toward both Democrats and Republicans about equally often in past elections.) That was the case in 2015, for instance, when Conservatives outperformed their polls by a net of 6 percentage points. There was an even worse error in 1992, when polls showed Labour narrowly ahead but instead Conservatives won in a landslide, making for a 9-point polling miss. That election gave rise to the term “Shy Tory Factor,” the idea that Conservative (Tory) voters were reluctant to declare their true voting intention to pollsters.

YEAR POLLING AVERAGE ACTUAL RESULT ACTUAL V. POLLS
2015 Con. +0.6 Con. +6.5 Con. +5.9
2010 Con. +7.9 Con. +7.2 Lab. +0.7
2005 Lab. +6.2 Lab. +2.9 Con. +3.3
2001 Lab. +14.2 Lab. +9.4 Con. +4.8
1997 Lab. +17.5 Lab. +12.8 Con. +4.7
1992 Lab. +1.5 Con. +7.6 Con. +9.1
1987 Con. +8.1 Con. +11.7 Con. +3.6
1983 Con. +20.3 Con. +15.2 Lab. +5.1
1979 Con. +5.9 Con. +7.2 Con. +1.3
1974 (Oct.) Lab. +9.2 Lab. +3.6 Con. +5.6
1974 (Feb.) Con. +2.9 Con. +0.6 Lab. +2.3
1970 Lab. +4.1 Con. +3.4 Con. +7.5
1966 Lab. +11.2 Lab. +6.0 Con. +5.2
1964 Lab. +1.5 Lab. +0.8 Con. +0.7
1959 Con. +3.2 Con. +5.6 Con. +2.4
1955 Con. +3.3 Con. +3.2 Lab. +0.1
1951 Con. +4.5 Lab. +0.8 Lab. +5.3
1950 Con. +0.7 Lab. +2.8 Lab. +3.5
1945 Lab. +6.0 Lab. +8.0 Lab. +2.0
Conservatives have recently outperformed their polls

For elections since 1974, results reflect Great Britain (England, Scotland and Wales) only and not Northern Ireland. Most UK pollsters have excluded Northern Ireland from their samples in recent years.

Source: Report of the Inquiry into the 2015 British general election opinion polls, Researchbriefings.parliament.uk

Exactly how strong the Conservative tendency to outperform their polls has been depends on where you measure from. Since 1992, Conservatives have beaten their final polling margin over Labour by an average of 4.5 percentage points, and have done so in all but one election. (That was 2010, when both Conservatives and Labour gained ground as Liberal Democrats’ support collapsed, but Labour slightly outperformed its polling margin against the Tories.) Go all the way back to 1945, however, and the average Conservative overperformance is just 1.8 percentage points and is not statistically significant.

Any extra support beyond what polls show would help May to stay out of the danger zone from a hung parliament. For instance, if you assume that Conservatives will benefit from a “Shy Tory Factor” and beat their polls by 3 percentage points, then what looks like an 7-point lead for May is really more like a 10point lead. Even acknowledging the high uncertainty in U.K. polling, a hung parliament would then only be an outside threat — using some very rough approximations, the chances of it would be somewhere in the neighborhood of 15 percent. By contrast, if the polls are taken at face value, the chances of it are more like 35 percent under the same assumptions.


If that’s the good news for May, here’s the bad news: The conventional wisdom is still pretty confident that she’ll win a majority, more so than the polls are. And the conventional wisdom is almost always wrong.

I’m not just being a trollish contrarian here. The conventional wisdom, at least as espoused by (i) betting markets and (ii) mainstream media coverage, has a remarkably poor track record in major elections around the world in recent years. In the U.K. last year, pundits and punters were irrationally confident of a “Remain” victory in the Brexit vote, even though polls showed it only barely ahead of “Leave.” In the U.S., they ignored how much the race had tightened in the final weeks of the campaign and data that showed Trump would likely do better in the Electoral College than the popular vote. In the French presidential election last month, the conventional wisdom was irrationally worried about a Marine Le Pen victory even though she trailed Emmanuel Macron by 20 to 25 percentage points. In fact, it was Macron who beat his polls, winning by 32 points.

These experiences have given rise to what I’ve called the First Rule of Polling Errors, which is that polls almost always miss in the opposite direction of what pundits expect:

The rule could potentially apply in the case of the U.K. election because there’s a gap between what the polls show and how the conventional wisdom assesses the race. Media coverage hasn’t been quite as much in lockstep as it was for Brexit or Trump, but the implication is usually that the “smart money” is on May winning a more comfortable victory than polls show. (YouGov has been excoriated by some London media outlets for daring to publish its model showing a hung parliament, for instance.) And betting market prices imply a Conservative win by 9 or 10 percentage points rather than their 7-point lead in the polling average. Since pundits expect Conservatives to beat their polls, the First Rule of Polling Errors would therefore predict that Labour would beat their polls instead. That wouldn’t necessarily yield a hung parliament or a Labour-led government, but it would make things awfully close.


So which idea — The Shy Tory Factor or the First Rule of Polling Errors — will prevail this time? Well, we’ll find out on June 8. But they have a complicated relationship with one another. Conservatives really do have a track record of beating their polls. But if everyone knows this and worries about it happening again — including pollsters — it could make the effect disappear or even reverse itself. My view, therefore — I’ll spend the rest of this article explaining my thinking behind this — is that the error is about equally likely to come in either direction. There’s a significant chance of Conservatives beating their polls again, but Labour is roughly as likely to do so.

Imagine that you move into a new apartment one summer and the thermostat is a little off. To actually get the room to feel like 72 degrees, you have to set the thermostat to 68. All of this is somewhat annoying, but at least you have a good rule of thumb: Take whatever number the thermostat shows and add 4 degrees to get the actual temperature. A lot of people are treating the U.K. polls in this way — as though they’re some sort of miscalibrated instrument that reliably misses in the same direction.

But suppose you ask your landlord — who is no mechanical genius but does the best he can — to fix the thermostat. Will it show the right temperature after that? Well, perhaps. But maybe the landlord replaces the wrong part and it doesn’t help at all. Or maybe the thermostat was working fine, but the air conditioner was broken. The landlord tinkers with it until it shows the right temperature while the A.C. is on, but it now overestimates the temperature when the heat is on instead. Under any given set of circumstances, the temperature could be too hot, too cold or just right.

This is what polling is like in the real world. If an indicator has historically had a bias, but human beings are aware of the bias and have an opportunity to correct it, it will often cease to be biased. Pollsters are under a lot of pressure to get the answer right, and they’re constantly tinkering with their methods. But there are a lot of moving parts, and there’s only one election every few years for them to test their methods upon. If they don’t make any changes, pollsters might duplicate a previous mistake. But they also might overcompensate for one-off circumstances that won’t replicate themselves again. A pollster who lowballed Democrats’ performance in 2008 and 2012 because they underestimated African-American turnout under Barack Obama might switch to a laxer turnout model for 2016, for instance, only to discover that Hillary Clinton didn’t turn out black voters like Obama did. There’s a risk of “fighting the last war” instead of looking ahead toward changes in the landscape.

In the U.S., the result of this series of measures and countermeasures is that the polls have not shown any long-term statistical bias toward Democrats or Republicans. They’ve been accurate in some years and not so accurate in others. But when they’ve missed, they’ve been about equally likely to miss in either direction.

This is also the case in Europeanwide polling. It’s often assumed that nationalist or right-wing parties tend to beat their polls, perhaps because (as is supposedly the case with Shy Tory voters) people are reluctant to declare their support for a “politically incorrect” party to pollsters. But over dozens of European elections over the past several years, there’s been no systematic tendency for nationalist parties to outperform their polls. Yes, it happens sometimes — such as in the 2015 election in Denmark. But the nationalist party has underperformed their polls almost exactly as often, as Marine Le Pen and the National Front did last month in France.

But if there’s no long-term bias in the polls in the U.S. or in Europe, what accounts for Conservatives so consistently beating their polls in the U.K.? I’ve done a lot of reading on this question and … I really don’t know. There aren’t a lot of great systematic explanations for the trend. However, there are lots of hypotheses about why the polls have been off in particular elections. In the U.K. context, some common ones include:

  1. “Shy” voters who don’t state their true voting intentions;
  2. Overly lax turnout models;
  3. Last-minute swings that come too late in the campaign to be captured by polls;
  4. Nonrepresentative samples or improper demographic weighting;
  5. Incorrect assumptions about undecided voters;
  6. Tactical voting (voters abandoning their top choice — often a minor party — for a party they think can win), and
  7. Pollster “herding” toward incorrect targets.

The British Polling Council and the Market Research Society issued a lengthy study after the 2015 general election, for instance, and concluded that the error in that election was mostly a result of Category No. 4. There wasn’t much evidence of actual “Shy Tories” in 2015, the report concluded; the problem wasn’t that Conservative voters were misleading pollsters but that there weren’t enough Conservative voters being polled in the first place. Among other issues, the polls were sampling too many highly informed voters, who were more likely to vote Labour; a similar problem (polls oversampling educated voters) led to Trump’s performance being underestimated in the U.S. There was also evidence of pollster herding in 2015 (Category No. 7): The final polls were within an unusually tight range, and at least one pollster suppressed a result that deviated from the consensus and showed the Conservatives winning big.

Other elections have come with different explanations for the polling miss, however. In 2010 — when the polls did well in capturing the Labour-Conservative margin but overestimated Liberal Democrats — the miss was initially attributed to a late swing (Category No. 3), perhaps exacerbated by tactical voting (Category No. 6). Although, this explanation is now disputed. The huge miss in 1992 really may have been because of Shy Tories (Category No. 1). In 1997, 2001 and 2005, Labour was predicted to win by large margins, but won by smaller margins instead. These were low-turnout elections, however, so incorrect turnout models (Category No. 2) might have been the main issue. The handling of undecided voters (Category No. 5) can also be a problem in lopsided elections such as these. Imagine that Labour is ahead 48 to 32 with 20 percent of voters undecided. In the U.S., that would be reported as a 16-point lead. But British pollsters don’t just let their undecideds be — they either drop from the sample or use some method to allocate them between the parties. If a pollster dropped the undecideds, the 48-to-32 Labour lead would instead be reported as 60-to-40 lead — a 20-point lead rather than 16.

Could it really be a coincidence that all these different errors in all these different elections have just so happened to underestimate Conservatives? Well, maybe. As I wrote earlier, the statistical significance of the Tory overperformance against the polls is somewhat dubious. (It also depends on where you draw the endpoints: It sounds impressive to say that Conservatives have outperformed their margin in six of the last seven elections, but less impressive to say they’ve gone 12 of 19.) It’s perhaps also noteworthy that while Conservatives have outperformed their polls in eight of nine elections when Labour was ahead in the pre-election polling average — but only four of 10 when Conservatives held the lead to begin with, as they do this year. In 1983, for instance, when the Tories entered the election with a 20-point lead, they wound up winning by “only” 15 points instead. The empirically derived rule of thumb “add a couple of points to Conservatives” could just as easily be “add several points to Conservatives if Labour is ahead in the polls, but don’t make too many assumptions otherwise.” Or it could even be “discount large leads in the polls, because parties tend to underperform them,” which would work against Conservatives this year.

PRE-ELECTION POLL LEADER POLLING AVERAGE ACTUAL RESULT ACTUAL V. POLLS
Conservatives Con. 5.7 Con. 5.4 Lab. 0.4
Labour Lab. 7.9 Lab. 3.6 Con. 4.3
On average, neither major party has outperformed polls when already ahead

Averages for all elections since 1945. For elections since 1974, results reflect Great Britain (England, Scotland and Wales) only and not Northern Ireland.

Source: Report of the Inquiry into the 2015 British general election opinion polls, Researchbriefings.parliament.uk


There would nevertheless be good reason to expect polls to lowball Conservatives again if pollsters were using the same methods they did in 2015. However, U.K. pollsters are fully aware of their error and have come to different conclusions about how to remedy it. Much to their credit, the pollsters are not herding this year; instead they show outcomes that range from a Margaret Thatcher-style Tory landslide to a hung parliament. These differences are not the result of random sampling error alone but instead reflect pronounced methodological and philosophical differences between the polls. Overall, U.K. pollsters are less traditional in their methods and more “creative” than U.S. pollsters are at trying out new methods, which could turn out to be a good thing or a bad thing. But they aren’t just taking the 2015 miss lying down.

The collective effect of these methodological changes is to show Conservatives with a larger lead than they would have otherwise. Again to their credit, U.K. pollsters often report multiple versions of their polls instead of just the final result; sometimes these are called “raw voting intention” and “headline voting intention.” A recent ComRes poll initially showed Conservatives with just a 4 percentage point lead based on respondents’ self-reported likelihood to vote (this is raw voting intention). After ComRes applied its turnout model and reallocated undecided voters, however, the Conservative lead grew to 12 points instead, which was the headline result.

On average among the various survey firms that published both results, the headline numbers have been 4 or 5 percentage points more Conservative-leaning than the raw version of the polls. By comparison, these adjustments moved the results by only 1 or 2 points toward Conservatives in 2015.

POLLSTER RAW VOTING INTENTION HEADLINE VOTING INTENTION
Survation Tied Con. +1
ICM Con. +2
Con. +11
ComRes Con. +4
Con. +12
Opinium Con. +4
Con. +6
Ipsos MORI Lab. +3
Con. +5
YouGov Con. +1
Con. +3
Kantar Public Con. +8
Con. +10
Average Con. +2.3
Con. +6.9
Pollster adjustments are shifting results toward Conservatives

Source: Huffington Post Pollster, IPSPS MORI

Are these various adjustments kosher? Or do some of them count as “unskewing”? I have certain approaches I like and others that I don’t, but it’s not really my place to say. My point is merely that pollsters are already attempting to compensate for their historical tendency to underestimate the Conservative vote. They might wind up overcompensating or they might wind up undercompensating. But it’s sort of double-counting to make an additional mental adjustment and to further inflate the Conservative vote above the methodological adjustments that pollsters are already making.

There’s also reason to be skeptical that the contrarian First Rule of Polling Errors (which would imply that Labour beats its polls) will apply in this election, however. The rule is mostly intended for cases in which pollsters are herding toward an artificial consensus, which may be influenced by the conventional wisdom. Instead, U.K. pollsters are experimenting with different methods and have been willing to show a wide range of results. The fact that British pollsters show such a diversity of outcomes is a reason to be more confident in them, not less.

To sum things up, I’d give the same advice that I pretty much always do on the eve of an election. Focus on the polling average — Conservatives ahead by 7 points — rather than only the polls you like. But assume there’s a wide range of outcomes and that the errors are equally likely to come in either direction. Given the poor historical accuracy of U.K. polls, in fact, the true margin of error on the Labour-Conservative margin is plus or minus 10 points. That would imply that anything from a 17-point Conservative win to a 3-point Labour win is possible. And even an average polling error would make the difference between May expanding her majority and losing it.