Category Archives: Press Releases

How Have The Polls Changed Since 2015?

Press Releases,

The performance of the polls in the 2015 general election left something to be desired. On average the final polls (that is, those whose fieldwork concluded no more than two days before polling day) put the Conservatives and Labour equal on 34%, when, in practice, the Conservatives proved to be on 38% and Labour on 31%. Indeed, throughout the election campaign the polls mostly suggested that the two largest parties were more or less neck and neck. Although the polls largely anticipated the performance of all of the other parties, including the SNP in Scotland, reasonably well, their misestimate of the performance of the two largest parties unsurprisingly gave rise to the widespread perception that the polls ‘had got it wrong’.

In the wake of this error, the British Polling Council and the Market Research Council appointed an Independent Inquiry under the Chairmanship of Prof. Patrick Sturgis to investigate why the inaccuracy arose and to make recommendations about how polls should be conducted and published in future. The Inquiry made the following recommendations about the future conduct of polls in its final report:

We recommend that BPC members should:

  1. include questions during the short campaign to determine whether respondents have already voted by post. Where respondents have already voted by post they should not be asked the likelihood to vote question.
  2. review existing methods for determining turnout probabilities. Too much reliance is currently placed on self-report questions which require respondents to rate how likely they are to vote, with no clear rationale for allocating a turnout probability to the answer choices.
  3. review current allocation methods for respondents who say they don’t know, or refuse to disclose which party they intend to vote for. Existing procedures are ad hoc and lack a coherent theoretical rationale. Model- based imputation procedures merit consideration as an alternative to current approaches.

The report also recommended that:

  1. Pollsters should take measures to obtain more representative samples within the weighting cells they employ.

and:

  1. Pollsters should investigate new quota and weighting variables which are correlated with propensity to be observed in the poll sample and vote intention.

In responding to the report the British Polling Council, inter alia, agreed the following motion:

The Council recognises the widespread public interest in how polls are conducted, and in ensuring that they are as accurate as possible. It also recognises the need to promote public understanding and informed discussion of how polls are conducted, especially at the time of a UK general election.

The Council thus resolves to commission for publication and launch in autumn 2019, a report that describes the methods being used by its members in their polls of UK election voting intentions and how these methods have changed since 2015. This report will be launched at a seminar at which individual member companies will present details of their current methodology.

This motion was, of course, written in the expectation that, given the terms of the Fixed Terms Parliament Act, the next general election would be held in May 2020. The decision of the Prime Minister, endorsed by the House of Commons, that a snap election should be held on 8 June 2017 has meant that the BPC has been unable to implement its motion by holding an event and publishing a report as it originally intended.

Even so, many people are inevitably still asking whether the polls have learnt from what happened in 2015 and thus can be relied upon to provide a more accurate estimate of Conservative and Labour support in the 2017 election. This report, published on the occasion of a seminar being held by the BPC in collaboration with the National Centre for Research Methods on how polls are being conducted in the 2015 election, is intended to help the reader answer that question for themselves. It provides for each BPC member that has so far been regularly conducting polls during the election campaign a summary of the principal changes they have made since 2015 to the way in which they conduct polls of voting intention.

It will be apparent from the reports that each company has responded to the lessons of the 2015 election in its own particular way, and that no two companies are conducting their polls in exactly the same way. However, two general themes emerge from the individual reports.

First, most companies have implemented one or more changes designed to improve their estimates of which voters are likely to make it to the polls and which not. In some instances, this has consisted of making greater effort to ensure that more potential non-voters are interviewed in the first place. A lot more effort is certainly being made to weight the interviews that are obtained so that known key patterns and differences in turnout and political engagement are reflected in the data upon which vote intentions are estimated. In many cases this endeavour has been facilitated by using the findings of the high-quality face to face random probability survey conducted after the 2015 election on behalf of the British Election Study, a step that was encouraged by the Sturgis Report.

Second, in some instances changes have also been made with a view to ensuring that the political balance amongst those who will make it to the polls is estimated accurately. Greater use is being made by some companies of weighting the data by how people voted in the 2015 election, while the opportunity is also being taken to ensure that the data are also reflective of how people voted in the EU referendum in 2016. At the same time, one of the lessons of that referendum, that sometimes people’s educational background matters to how they vote, means that some companies are now also weighting their data by that characteristic too.

The BPC is not responsible for how opinion polls are conducted; that is the responsibility of each of its members individually. The BPC’s role is to promote transparency about how polls are conducted so that journalists, commentators and the general public can make their own informed decisions about whether to believe the polls or not. We hope that they find it useful in forming their own judgement as to whether the polls have learnt the lessons of the 2015 election.

John Curtice
President, British Polling Council

ComRes

ComRes has continued to strive to improve our methodology and accuracy since the 2015 general election. We conducted an internal review following the election and indeed paused the publication of Voting Intention polls for a period in order to enable us to review our approach properly.

Immediately after the 2015 election we introduced a new model of turnout in order to map voting intention more accurately. The Voter Turnout Model was developed through a multiple regression analysis of ward level demographic and turnout data. Although General Election results do not provide a count of who exactly turns out to vote, this analysis allows us to develop a robust estimate of that phenomenon.

Through a series of experiments testing the relationships between different demographics and turnout, ComRes consistently found that the two most important drivers of turnout are age and social grade. Using regression analysis, we developed a weighting matrix which predicts turnout in different demographic groups more reliably than self-claimed likelihood to vote and the resulting factor weights are now used in weighting our data. The value of the model rests on the fact that the variation in turnout by demographic group is relatively consistent from one General Election to the next.

We have subsequently revisited the polls we conducted during the 2015 election campaign to measure the impact of applying this model. If we had used this method in 2015 and made no other adjustments to our methodology, our final poll would have had a 5 point Conservative lead over Labour. This pattern is repeated in each poll for which a comparative analysis has been conducted.

We are confident that this is the most important change we can and need to make to our methodology for estimating vote intention. Although we, along with the rest of the industry, overstated Labour and underestimated Conservative support in 2015, we got the other party vote shares almost spot-on. Detailed investigations, including of the relative merits of our polls conducted by phone and those conducted online (we did both in 2015), lead us to conclude that estimating the pattern of turnout correctly rather than sampling is the critical factor required to improve the accuracy of our polls. At this election, all of our polls are being conducted online.

ICM Unlimited

ICM’s polls are being conducted in 2017 in a number of different ways from how they were conducted in 2015. All polling is now conducted via an online panel, whereas in 2015 it was conducted via telephone using random digit dialling. At the same time a variety of other changes to the way in which the sample is obtained and the resultant data are weighted and adjusted have been made. These are as follows:

  1. Sample Size. All polls now typically contain at least 2,000 respondents rather than 1,000.
  2. Quotas. In 2015 quotas (i.e. targets for who should be interviewed) were set for a range of demographic factors such as age, gender, and social grade. Now they also include for which party people say they voted at the last general election.
  3. Turnout. In 2015 respondents were asked how likely they were to vote on a scale from 1 to 10. In estimating vote shares for each party each respondent was weighted according to their answer to this question. Thus, for example, respondents who said they are 5/10 likely to vote were given a weight of 0.5. In addition, those who said they did not vote at the previous general election were further downweighted by a half. In 2017, the data are weighted so that the anticipated level of turnout for those in different age groups and social grades matches the reported level of turnout for each combination of age and social grade in 2015 as measured by a variety of sources.
  4. Weighting by Interest in Politics. In 2017 the interview data are being weighted so that the distribution of reported interest in politics matched that obtained by the face to face random probability survey conducted on behalf of the British Election Study after the 2015 general election. No such weighting was applied in 2015.
  5. Weighting by Past Vote. In 2015 the interview data were weighted to a target for each party that was 80% based on the actual result of the last election and 20% to the distribution of past vote (prior to weighting) in the last 20 polls conducted by ICM. The latter part of this formula was intended to reflect the fact that some respondents may have misreported how they voted at the last election. In 2017, however, the data are weighted so that the distribution of reported vote at the last election fully matches the actual result.
  6. Imputing Vote Intentions of those who do not indicate how they will vote. In 2015 those who failed to state a vote intention but indicated for which party they voted in 2010 were assumed to be most likely to vote for the same party again and were added to that party’s tally at a value of 0.5. In 2017 the same procedure is being applied again, except that those who said they voted Conservative or Labour at the last election are being added at a value of 0.75. In addition, the result of this adjustment is now also applied to those who not only fail to state a vote intention for the current election but also do not state how they voted at the last election, except that this group is estimated to be 20% more likely to vote Conservative and 20% less likely to vote Labour.

Ipsos MORI

Over the past two years Ipsos MORI has carefully reviewed its approach on the analysis and reporting of its opinion polls. In the 2015 General Election we believe we sampled too many engaged, higher educated voters, which helps to explain our over-estimate of the Labour vote share by 3.8 percentage points. We did not under-represent the total proportion of Conservative voters, but rather did not interview enough non-voters. Compared with our results in previous elections, we also found a disproportionate number of Labour-leaning voters saying they would vote, some of whom we believe instead chose to abstain.

In light of this we have made three methodological changes since that election: 1) weighting by newspaper readership, 2) setting quotas and weights for educational attainment, and 3) making changes to our turnout filter. We outline these modifications below.

In 2015, after the General Election, we began weighting our samples by newspaper readership, which went some way towards offsetting our initial sample bias towards those who are more politically engaged. (In particular, these weights bring down the proportion of broadsheet readers and increase the proportion of tabloid readers.) We believe this change would have made us more accurate in the 2015 General Election; however, it did not completely overcome the issue of disproportionally interviewing too many graduates and not enough individuals without qualifications.

To compensate, in 2016 we added educational attainment to both our interview quotas and weighting scheme. Adding these elements to our polling approach we believe has made our sample more representative, and indirectly accounts for level of political engagement as both educational attainment and newspaper readership are strongly correlated with interest in and knowledge of politics.

After the 2015 General Election we also introduced a new turnout filter to our published monthly Political Monitor polls, to help mitigate the issue of interviewing individuals who say they will vote but in fact do not show up to the polling station – something which was more of a challenge for us in 2015 than in previous elections. Previously, we have relied simply upon (i) excluding unregistered voters and (ii) using a 1-to-10 likelihood of voting question, taking those responding ’10’ (absolutely certain to vote) as voters and everybody else as non-voters.

Our new turnout filter includes the use of an additional question measuring general election voting frequency (Which of the following best describes how often you vote in General elections? I never vote/I rarely vote/I sometimes vote/I usually vote/I always vote/It depends/Don’t Know). Respondents are now excluded from the headline figure if (a) they say they are not registered to vote in the election or don’t know; or (b) they rate their likelihood of voting at below 9, or don’t know; or (c) they say they “never”, “rarely” or “sometimes” vote in general elections, or don’t know. This filter would also have made us more accurate in the 2015 General Election.

Kantar Public UK

Note Kantar Public polls were previously published under the name TNS UK. We have made the following changes to our polling methods since the 2015 election.

Likelihood of Voting

  • We have developed a new LTV (likelihood to vote) model. The likelihood that each respondent will vote in the General Election is estimated based on respondents’ stated intention to vote, their age and whether they voted in the last general election. This likelihood is used to weight the contribution that each respondent makes to the tally of estimated vote intentions for each party. The model was developed using data from respondents who participated in TNS polls immediately prior to the 2015 general election and immediately afterwards. In addition, a further small adjustment has been made to correct for over-claiming to have voted, based on analysis of the face to face random probability survey conducted after the 2015 election for the British Election Study.

Weighting

  • We have modified the way in which we weight our data by age to ensure that the right proportion of the weighted data is aged 70+.
  • Education is now included in our weighting targets to ensure that we do not have too many respondents with a degree.
  • To ensure that our sample is not too politically engaged the data are now also weighted so that the overall proportion who are expected to vote (estimated via our LTV model) matches our estimate of the likely turnout. This estimated turnout is based on the relationship between the likely turnout in our 2015 data and the actual turnout in the 2015 general election.

Questionnaire

  • In the short campaign, we are now asking whether or not people have voted by post, and if they have already voted we do not ask the likelihood to vote question.

Treatment of “Don’t know” and “Refused” at the voting intention question

  • We now ask a ‘squeeze’ (or follow-up) question of those who plan not to vote /don’t know how they will vote / refuse to say who they will vote for. This question asks respondents which party they prefer, and those naming a party in response to this question are added to the relevant tally.

    In all of the polls we have published so far those that do not provide a response to the initial question on voting intention or to the squeeze question are excluded from our final voting intention figures. However, we think it is unlikely that such voters will distribute their support in the same way as those who do declare a vote intention, and we therefore plan to develop an imputation approach to use in our final polls.

Opinium

The polls conducted by Opinium at this election differ from those undertaken at the time of the 2015 election as follows:

  1. Our estimates of party support are now based on those who say they are 10/10 certain to vote when they are asked to indicate how likely they are to vote on a scale from 0 to 10. In 2015 likelihood of voting was ascertained by asking respondents whether they definitely/probably would/would not vote and our estimates of party support were based on those who said they would definitely vote. The new procedure is somewhat more restrictive in terms of the proportion of sample who is regarded as likely to vote.

  2. We have changed the weighting that is applied to ensure that our sample is politically representative.

    Prior to our 18 April 2017 poll, our results weighted by EU referendum vote and party propensity as well as by standard demographic characteristics.

    Our party propensity model was based on asking respondents their likelihood of ever voting for each party on a 1-10 scale and then assigning them to a group (e.g. “Conservative – lean left” or “Labour – loyalist”). Targets for the proportion of the sample that should belong to each group were based on a rolling average that incorporated (i) the proportions found in each group in polling that was conducted immediately after the most recent general election (including, the EU referendum) after the data had been weighted to the actual result, and (ii) the proportions found in each group in subsequent polls when only demographic weighting had been applied.

    Party propensity was therefore designed to include a measure of weighting by past general election vote but with the weight that this carried in the generation of the propensity group targets diminishing the further we were removed from the last election.

    However, it had become increasingly clear that this procedure was no longer sufficient on its own. For example, in our poll dated 11 April 2017 it had the effect of ensuring that the poll contained equal proportions of respondents who said they voted Labour and Conservative in 2015, with both groups comprising 28% of the sample. We have therefore implemented a stricter 2015 past vote weight that ensures that the proportion who say they voted for each party in 2015 matches the actual outcome of that election. However, our weighting does not attempt to bring the proportion of respondents who say that they voted in line with the actual turnout last time around. Our data are still also weighted by EU referendum vote.

  3. In calculating the distribution of vote intention we are now also downweighting by a half those who say they will vote for a party but also say that they disapprove of that party’s leader.

ORB International

ORB International did not publish polls of voting intention during the 2015 general election, but is doing so during the 2017 election. An outline of our methodology in 2017 is given below:

Panel

Our polls are being conducted via an online panel of potential respondents. This uses a stratified sampling technique to obtain a population that best represents the entire population being studied. For any particular poll, the email addresses of panel members are selected at random using interlocking quotas of age, gender, social grade and region. The size of each quota category takes into account the predicted response rate.

Weights

The interview data are weighted to be demographically representative on the following variables according to targets taken from the National Readership Survey:

  • Gender
  • Socioeconomic grade
  • Tenure
  • Work Status
  • Cars in household
  • Foreign holidays
  • Geographic region

In addition, weights for age and education level are applied so that the size of each age/education group reflects the proportion those who actually voted in 2015 who belonged to that group. The targets for these weights are derived from the race to face random probability survey that was conducted after the 2015 election on behalf of the British Election Study.

Finally, in calculating vote intention a weight for the likelihood that a respondent will vote is applied according to their answers to the following question.

And once again thinking to the next general election on June 8th, on a scale of 0-10 where 0 is certain not to vote and 10 is certain to vote, how likely is it that you would go out and vote?

Thus, for example, a respondent who gives an answer of 5 counts as 0.5 in calculating the tally of vote intentions.

1 2 3 4 5 6 7 8 9 10  
Certain not to vote                 Certain to vote Don’t know

Reported Voting Intention

Our reported voting intention takes the weighted data and includes all those who name a party to provide a final indication of voting intention.

Panelbase

We are weighting our polls not only by how people voted in the 2015 general election but also by how people voted in the EU referendum in 2016. Otherwise, to date, we have not made any radical change to the online polling methodology used in 2015. However, for our final polls, we are considering using data from the face to face random probability survey conducted after the last election for the British Election Study in order to estimate likelihood of voting.

Survation

Survation are conducting polls both by phone (for Good Morning Britain) and online (for the Mail on Sunday).

Our phone polls are being conducted using the same approach as that used in the final poll that we conducted before the 2015 election which put the Conservatives on 37% and Labour on 31%, but which was not published in advance of the election count. Data that links phone numbers to lifestyle information about the households in question is used to generate a stratified random sample of respondents.

Our internet polls are also being conducted using the same methodology as in 2015.

YouGov

We have made the following changes to our methodology since the 2015 general election.

Overview

YouGov conducted an in-house review of our election polls (available online here:). This came to similar conclusions to the BPC/MRS inquiry – that the 2015 polling error was primary down to sampling issues. Specifically, we felt that the sample was skewed towards people who were too interested in politics, particularly among young people.

Our changes since then have been primarily aimed at addressing this issue through improvements to the composition of our panel of potential respondents and changes to quotas and weights.

Panel Recruitment

Since 2015 we have focused our efforts at recruiting people into our panel of potential respondents more specifically upon those who belong to groups that were under-represented in our samples during the 2015 election, particularly those with a low level of interest in politics, younger panellists with low educational qualifications, and those who did not vote at the previous general election.

Sample quotas and weighting targets

We have changed upper age band that we use both in determining who should be asked to participate in a poll (quota) and in subsequently weighting the data we obtain so that it includes only those aged 65 and over rather than those aged 60 and over. This has increased the proportion of those aged 65 plus who are represented in our samples and corrects an over-representation of those aged between 60 and 64.

We have dropped our previous quotas and weights by newspaper readership which, with falling print readership, we found was no longer effective for enough of the sample.

We have added quotas and weights by highest educational qualification, calculated separately for each age group, in order to correct an over-representation of graduates, particularly among younger respondents.

We have replaced political weighting and quotas by party identification with recalled past vote, calculated separately for each region. This has given us greater control over the number of people in the sample who did not vote in 2015, while weighting separately for each region has improved the political representativeness of our samples in Scotland and London.

We have now also set quotas and weight our data by the amount of attention the respondent pays to politics. The targets for these quotas and weights are based upon the face to face random probability survey conducted after the last election on behalf of the British Election Study.

Since 2016 we have also weighted by reported EU referendum vote, collected from our panel members shortly after the referendum.

Likelihood to vote

Finally, we have changed our likelihood to vote model. Respondents are still asked to say on a scale from zero to ten how likely they are to vote in the general election, and the contribution they make to the estimate of vote intentions reflects their answer to this question. Thus, for example, respondents who say they are 5/10 likely to vote are given a weight of 0.5. However, this contribution is now reduced by a half if the respondent says they did not vote at the 2015 general election.

BPC Comments on PRCA enquiry into political polling and predictions

Press Releases

The British Polling Council welcomes the interest of the Public Relations and Communications Association (PRCA) in the work of the polling industry, with its announcement that it is to hold an enquiry that, according to PR Week, ‘will address trust and confidence in political predictions following the May 2015 General Election, the UK referendum result in June, and the US Presidential Election result earlier this month’.

The BPC would like to highlight the extensive investigations that have and continue to be made into the performance of the polls. The evidence and conclusions of these reviews will, the BPC believes, provide a valuable source of material to assist the PRCA in its enquiry. These reviews set out the challenges facing the opinion polling industry, covering both what they are getting right as well as the premise as to whether (and where) the ‘pollsters got it wrong’ in the Brexit referendum and the US Presidential election.

Following the undoubted underestimation of the Conservative vote and over-estimate of the Labour vote at the 2015 general election, the BPC, along with the Market Research Society (MRS), sponsored a wholly independent inquiry under the chairmanship of Prof. Patrick Sturgis. That inquiry published an extensive 115 page report and made a series of recommendations that the BPC is currently implementing. Meanwhile, the BPC and MRS are sponsoring a public seminar on the performance of the polls in the EU referendum that will be held at the Royal Statistical Society on 8th December.

On the other side of the Atlantic, the American Association of Public Opinion Research (AAPOR) announced last spring that, in line with its practice after previous presidential elections, a panel under the chairmanship of Courtney Kennedy will analyse and report on the performance of the polls in the US Presidential election. The Panel is due to report by May 2017.

Pollsters in the UK are far from complacent about their performance in recent ballots. However, it is to be hoped that the PRCA’s enquiry will add to rather than duplicate the existing attempts that are being made to assess and improve the methodology of the polling industry. The BPC would certainly welcome an opportunity to advise the enquiry of the efforts that have been made to date.

It is, of course, the case that most of the final polls conducted by BPC members pointed to a victory for Remain rather than for Leave. But in most cases they also suggested that the outcome would be very close. Meanwhile, of the 34 polls of voting intentions conducted during the four week official campaign period between 27 May and 22 June, 17 put Leave ahead, 14 Remain while three suggested the two sides were tied. The evidence from the polls did not point unambiguously to a Remain victory or the simple assertion that the pollsters ‘got it wrong’.

Meanwhile in the United States it is all too often forgotten that, as the nationwide polls anticipated, Hillary Clinton won more votes than Donald Trump. At present she leads in the nationwide vote by around two points. This compares with an average lead for Ms Clinton in the final polls as calculated by Real Clear Politics of 3.2 points. If, when the final tally is available, the nationwide polls are shown to have overestimated the Democrat candidate’s lead by just over one point, this will be lower than the average error in polls of presidential elections between 1968 and 2012 of two points.

While the record of the polls in both the EU referendum and the US Presidential election was certainly not perfect there appears to be a widespread misperception that in both cases the polls pointed to a clear winner. They did not.

We therefore hope that the findings of these reviews into the opinion polls will provide a firm evidence base for the PCRA as it starts its enquiry and the BPC welcomes the opportunity for more dialogue as the review takes shape – including the question of how the findings of opinion polls can be better communicated to the public at large.

Performance of the polls in the EU referendum

Press Releases,

Seven member companies issued ‘final’ polls of voting intentions in the EU referendum. While no company forecast the eventual result exactly, in three cases the result was within the poll’s margin of error of plus or minus three points. In one case Leave were correctly estimated to be ahead. In the four remaining cases, however, support for Remain was clearly overestimated. This is obviously a disappointing result for the pollsters, and for the BPC, especially because every single poll, even those within sampling error, overstated the Remain vote share.

Polling of voting intentions in the referendum was a unique event, and the lessons of this experience will not necessarily be transferable to polling of future general elections. Consequently, the BPC does not believe that a full-scale independent inquiry, similar to that set up after the 2015 General Election, would justify the large amount of time involved. However, the Council will be asking all its members to look carefully at the methods that they used in the referendum and to report their findings, and will consider inviting an external reviewer to consider the methods that were used.

BPC President John Curtice commented, ‘Polling of voting intentions in this referendum was always going to be a difficult exercise. Yet their central message – that this looked like a close referendum that neither side could be sure of winning – proved prescient. Nevertheless, there will be an obligation on members to try and establish why there was a tendency to overestimate support for Remain, and the Council will report later this year on why that appears to have been the case.’

The following table shows the results of all polls conducted by BPC members that had at least some fieldwork within the last 4 days of the campaign.

Remain Leave error on remain Method Sample Size Fieldwork
ORB 54 46 +6 telephone 877 14-19 June
Survation 51 49 +2 telephone 1003 20-June
ComRes 54 46 +6 telephone 1032 17-22 June
Opinium 49 51 +1 online 3011 20-22 June
YouGov 51 49 +3 online 3766 20-23 June
Ipsos MORI 52 48 +4 telephone 1592 21-22 June
Populus 55 45 +7 online 4740 21-22 June
RESULT 48 52
Average error 4.3

As well as the pre-election polls there were three published on-the-day polls, by BMG, YouGov and Ipsos MORI. As they combine people who are reporting their actual vote and others who are reporting their voting intention they are treated separately from the pre-election polls above. The YouGov poll interviewed 4,772 people online and produced a result of Remain 52% Leave 48%. Ipsos MORI interviewed 546 people by telephone and produced a result of Remain 54% Leave 46%. This is a smaller sample size than their standard published polls as it was not intended as a standalone on-the-day poll, and only presented as a day-by-day comparison drawing on their final poll. BMG interviewed 2,000 people online with a declared result of Remain 46% Leave 41% and Don’t know 13%.

The ORB and Ipsos MORI polls covered only Great Britain. All the other polls above covered the whole of the UK.

British Polling Council member TNS-BMRB also conducted a poll within the fieldwork period above. However, they took the decision not to remove and reallocate in any way those who said they were undecided, or that they would not vote, and so no calculation of error is possible. For the record, their final poll was conducted online from 16-22 June, with a sample size of 2,320, and the results were Remain 41% Leave 43% Undecided/would not vote 16%

In the interests of completeness, British Polling Council member BMG Research conducted two polls, one telephone and one online, that finished over a week before polling day, and which are therefore not included in the table above. Fieldwork was conducted from 10-15 June, with sample sizes of 1,043 (telephone) and 1,468 (online). The results were Remain 53% Leave 47% (telephone) and Remain 44% Leave 56% (online).

British Polling Council member LucidTalk conducted a poll solely in Northern Ireland. Fieldwork was conducted online from 15-17 June, with a prediction of Remain 57% Leave 43%. As the actual Northern Ireland result was Remain 56% Leave 44% this represents an error of only 1% in its estimate of the vote.

Performance of the Opinion Polls in the 2016 Local and Devolved Elections

Press Releases,

The following tables compare the results of opinion polls of voting intentions for the elections that were held on May 5th in London, Scotland and Wales with the final outcome. A poll is included if its fieldwork was conducted wholly or mostly in the seven days before polling day (May 5th), and it was the final poll published by the pollster concerned. (This is a slightly longer period than is normally used in BPC reports on the performance of the polls in general elections — the period has been lengthened because there were very few polls conducted in the last few days.)

It should be noted that all of these polls were conducted online, with the exception of the Survation poll in Scotland, which was conducted by telephone.

London Mayor 1st Preference Vote

Company / Fieldwork Dates Con Lab LibDem UKIP Green Other
Opinium 26.4–1.5.16 35 48 4 5 5 3
TNS 26.4–3.5.16 33 45 7 5 4 5
ComRes 28.4–3.5.16 36 45 6 4 6 3
YouGov 2–4.5.16 32 43 6 7 7 5
RESULT 35 44 5 4 6 7

London Mayor After Redistribution of 2nd Preferences

Company / Fieldwork Dates Con Lab
Opinium 26.4–1.5.16 43 57
TNS 26.4–3.5.16 43 57
ComRes 28.4–3.5.16 44 56
YouGov 2–4.5.16 43 57
RESULT 43 57

London Assembly — Constituency Vote

Company / Fieldwork Dates Con Lab LibDem UKIP Green Other
YouGov 2–4.5.16 30 44 7 11 7 1
RESULT 31 42 8 8 9 3

London Assembly — List Vote

Company / Fieldwork Dates Con Lab LibDem UKIP Green Other
YouGov 2–4.5.16 29 39 8 11 9 5
RESULT 29 40 6 8 8 3

Scotland — Constituency Vote

Company / Fieldwork Dates Con Lab LibDem SNP Green Other
Survation 1–2.5.16 19 21 7 49 5
YouGov 2–4.5.16 19 22 7 48 – 4
RESULT 22 23 8 47 – 1

Scotland — List Vote

Company / Fieldwork Dates Con Lab LibDem SNP Green Other
Survation 1–2.5.16 20 19 8 43 7 4
YouGov 2–4.5.16 20 19 6 41 9 5
RESULT 23 19 5 42 7 4

Wales — Constituency Vote

Company / Fieldwork Dates Con Lab LibDem PC UKIP Other
YouGov 2–4.5.16 21 33 8 19 16 4
RESULT 21 35 8 21 12 3

Wales — List Vote

Company / Fieldwork Dates Con Lab LibDem PC UKIP Other
YouGov 5.5.16 20 31 6 20 16 8
RESULT 19 31 6 21 13 3

YouGov also conducted on polling day an exercise in which it recontacted a sample of those who it had interviewed previously in order to ascertain how they had or intended to vote. This produced the same result for the Constituency vote, and very similar figures for the list vote: Con 19%, Lab 30% LibDem 6%, PC 21%, UKIP 16% and other 8%.

BPC Inquiry Report

Press Releases

The British Polling Council welcomes the publication today of the report of the Independent Inquiry into the performance of the polls in the May 2015 general election. It is deeply grateful to the Chair, Prof. Patrick Sturgis, and his colleagues for their forensic and thorough analysis of why the polls underestimated Conservative and overestimated Labour performance at that election.

The principal objective of the Council is to promote transparency in the reporting of opinion polls, and its members are required to adhere to a set of rules designed to achieve that objective. The Inquiry’s report makes a series of recommendations for changes to those rules.

The Council has resolved that so far as some of those recommendations are concerned, the necessary changes to its rules should be made with immediate effect. Other of the Inquiry’s recommendations require preparatory work be undertaken before they can be implemented, and the Council has agreed that that work should be put in train in the expectation that the consequent rule changes can be introduced early in 2017.

Specifically, the Council has agreed to implement immediately rule changes that will (i) require greater transparency about how polls have been weighted, (ii) specify what changes, if any, have been made since a company’s previous published poll in how the data have been weighted or otherwise adjusted, and (iii) place an obligation on members to supply to any inquiry or committee that has been established by the BPC the micro data set for any poll in which that inquiry or committee has an interest.

Meanwhile, the Council has agreed that work should be undertaken to develop (i) an industry-wide method for calculating the confidence limits associated with a poll’s estimate of a party’s share of the vote, and (ii) an industry-wide approach to calculating the statistical significance of the change in a party’s estimated vote share since a company’s previously published poll. Rules that will require members to publish these calculations for any poll of vote intentions will be introduced once this work is completed. At the same time the Council will also consider how best to respond to the Inquiry’s recommendation that members should register with the BPC the fact that they are undertaking a poll.

The Inquiry also makes a series of recommendations for changing the way in which polls are conducted. It will be for individual member companies to decide how best to take these forward. At the same time, however, the Council is aware of the need to show that due note has been taken of those recommendations and to make clear what changes have as a result been made to the way in which polls are conducted. The Council has thus also resolved that (assuming the next election is held in May 2020) it will issue a report in the second half of 2019 that describes the methodology that is being used its members in the run up to the next election and how this methodology has changed since 2015.

Prof. John Curtice, President of the British Polling Council, said, ‘The Inquiry has undertaken what was an important but demanding task in a timely and professional fashion. I am confident that all those with an interest in understanding the difficulties that beset the polls in 2015 will find its report an illuminating and profitable read. The Council now wishes to ensure that its work is put to best use so that the transparency and accuracy of opinion polls is enhanced in future.’

Notes to Editors

  1. The British Polling Council (BPC) is an association of polling organisations that publish polls. The objectives of the Council are to ensure standards of disclosure that provide consumers of survey results that enter the public domain with an adequate basis for judging the reliability and validity of the results.
  2. The Inquirys Report is embargoed until 0001 hours on 31 March, and will be available online at National Centre for Research Methods (NCRM)
  3. For further information, please contact a member of the BPC’s Management Committee:
    • Simon Atkinson: 07791-680 770
    • Nick Moon: 07770-564 664
    • John Curtice 07710-348 755

British Polling Council Welcomes Unveiling of the Provisional Findings of Polling Inquiry

Press Releases

The British Polling Council (BPC) welcomes the unveiling today of the provisional findings of the Polling Inquiry on why the polls overestimated Labour and underestimated Conservative support in last May’s general election. Though established shortly after the last election by the BPC in collaboration with the Market Research Society, the Inquiry has been working wholly independently of its sponsors under the chairmanship of Prof. Patrick Sturgis of the University of Southampton. The Council is deeply grateful to Prof. Sturgis and his colleagues for the work they have undertaken so far and looks forward to receiving their final report in March.

Prof. John Curtice, President of the British Polling Council, said, ‘Today’s unveiling will provide the polling companies and everyone else with an interest in its work to hear and respond to the Inquiry’s initial conclusions. It hopes that this process will prove helpful to the work of the Inquiry and ensure that its final conclusions are based on the strongest possible body of evidence.’

Notes to Editors.

  1. The British Polling Council (BPC) is an association of polling organisations that publish polls. The Council promotes standards of disclosure that are designed to provide consumers of survey results that enter the public domain with an adequate basis for judging the reliability and validity of the results. Most of the companies that conducted polls of voting intention at the 2015 UK general election are members.
  2. The Polling Inquiry is presenting its provisional conclusions about the performance of the polls at a meeting held at the Royal Statistical Society this afternoon. Further details are to be found at the NCRM — The Inquiry into the Failure of the 2015 Pre-election Polls:.
  3. The terms of reference of the Polling Inquiry can be found at NCRM — BPC/MRS Inquiry into the Performance of the Opinion Polls at the 2015 General Election, and its membership at NCRM — Panel Membership. The Inquiry is due to submit its final written report to the BPC and the Market Research Society at the beginning of March.

British Polling Council Announces Findings in Respect of Complaint by Mr Dominic Cummings

Notice, Press Releases,

The British Polling Council (BPC) is today announcing its findings in respect of a complaint from Mr Dominic Cummings of Vote Leave about a survey of members of the CBI conducted by YouGov in June and July 2013.

The BPC’s principal aim is to ensure that its member companies are as transparent as possible about the methods that they use to conduct their polls. It does not adjudicate on the merits of the methodologies that are used in polls — and consequently it has not done so in this case. Our findings do not represent in any way a judgement on the merits of the methodology used by YouGov in the CBI poll.

Under its Rules, any complaint to the Council is in the first instance considered by the officers. If they determine that the Rules may not have been fully observed, the matter is drawn to the attention of the member company in question. Should the company agree to take any necessary action, the matter is then considered to have been resolved.

As indicated in a press release issued by the BPC on 2 November, because his initial view on the complaint had inadvertently become publicly known, the Secretary of the BPC, Mr Nick Moon, took no part in the handling of this complaint. It has been handled throughout by the President of the BPC, Prof. John Curtice, and the BPC Committee Member, Mr. Simon Atkinson.

The officers took the view that the complaint made two points that were relevant to the Council’s Rules.

  • That YouGov had failed to make clear whether and how the data in the survey had been weighted.
  • That YouGov had not provided an adequate explanation of the sampling procedures that had been used to conduct the survey.

On the first, the officers took the view that there were not sufficient grounds to uphold the complaint. On the second the officers noted that YouGov had supplied details of the sampling procedures to Vote Leave on request, but that this information was not publicly available on its web site. While YouGov’s website does provide a full description of how they conduct their regular polls amongst members of the general public, that statement did not cover this particular survey.

YouGov have corrected this omission and appended to the detailed tabulations of the CBI poll that are available on its website an explanation of the sampling procedures that were used to conduct the poll. Accordingly, the matter is now considered to be resolved.

Notes to Editors

  1. The BPC does not usually issue a press statement in respect of complaints that are made to it. It has done so exceptionally on this occasion because of the media publicity and interest that surrounded the original lodging of Mr Cummings’ complaint.
  2. The Objects and Rules of the British Polling Council can be found at Objects And Rules
  3. The detailed tabulations for the CBI poll can be found at YouGov / CBI Survey Results
  4. The methodological statement that has been appended to these tables reads as follows:

This was a survey of members of the CBI.

A list of members was given to YouGov by the CBI. All were invited to complete the survey, both by sending a letter and a paper questionnaire to their business address and by sending an online link to the survey by email.

To maximise the response rate the survey was left open for completion for a period of six weeks (from 13th June to 30th July 2013). During that time non-respondents were sent two reminder emails and contacted by telephone to encourage their participation.

The final number of responses was 415.

Statement re Complaint by Mr Dominic Cummings

Press Releases

The BPC today received a complaint from Mr Dominic Cummings about a poll conducted by YouGov for the CBI.

On receipt, any complaint is initially examined by the three principal officers of the British Polling Council, Prof. John Curtice (President), Mr Nick Moon (secretary) and Mr Simon Atkinson (committee member).

Unfortunately in expressing an initial provisional view about the complaint to his fellow officers Mr Moon copied Mr Cummings in to the relevant email and in so doing used some unnecessarily offensive language. For the latter Mr Moon has apologised unreservedly to Mr Cummings.

However, Mr Cummings appears to have taken the email as the determination of the BPC on his complaint and chosen to make Mr Moon’s provisional views public knowledge. In this Mr Cummings was clearly mistaken. The BPC has not made any determination on the matter, and apologises to YouGov and the CBI for its role in any unnecessary embarrassment that has been caused.

In view of this incident, the initial determination on Mr Cummings complaint will now be made by Prof. Curtice and Mr Atkinson alone.

@BritPollingCncl

British Polling Council opposes Bill on Regulating Opinion Polls

Press Releases, ,

The British Polling Council (BPC) urges the House of Lords to reject the private members bill on the regulation of opinion polls that is being presented today by Lord Foulkes.

The Bill proposes that an authority be established that would regulate polls of voting intentions for all elections and referendums in the United Kingdom. The authority would be empowered to specify approved ways for selecting who should be interviewed, how the questions in polls should be worded and to ban the publication of voting intention polls during an election campaign.

Who is interviewed by a poll and how the questions it asks are phrased are important issues. How any poll has addressed them should always be clearly stated, as the rules of the BPC require. But they are not issues that are susceptible to straightforwardly ‘right’ or ‘wrong’ answers. Professional researchers can and do disagree about how polls should be conducted and how they should be worded. They regularly experiment and test alternative and new ways of doing polls and asking questions in order to improve their methods. Imposing regulatory standards would put at risk the experimentation and competition that are essential to improving the ways in which polls are conducted.

Banning the publication of polls during an election campaign would not mean that polls were not conducted. It would simply mean that access to their results would be confined to those who could afford to pay for polls, such as the banks and the political parties, or who knew where to find the results on an overseas website. Only the ordinary voter, who is meant to be central to the democratic process, would be left out of the loop. It could potentially open the way for politicians to claim that their private polling showed them ahead, regardless of what their polling actually showed, or indeed whether it existed at all.

Doubtless many people feel that in underestimating the Conservative vote and overestimating Labour’s, the polls provided unhelpful misinformation during the recent general election campaign. That is why the BPC has established an independent inquiry into why the polls were wrong and how their conduct might be improved in future. Indeed, the first meeting of that inquiry, at which BPC members will be presenting – in public – their initial findings as to what went wrong, is being held today.

Professor John Curtice, President of the British Polling Council, said, ‘What is needed now is a critical and open appraisal of where the polls went wrong, not the heavy hand of regulation that in attempting to impose common standards would make it more likely that the polls all get it wrong again in future. As any economic forecaster knows too well, forecasting how people will behave is always a difficult enterprise. No-one has yet suggested that, despite their many errors, economic forecasting should be regulated, and it is not clear why attempting to anticipate how people will vote should be treated any differently.’

Notes for Editors:

  • The British Polling Council (BPC) is an association of polling organisations that publish polls. The objectives of the Council are to ensure standards of disclosure that provide consumers of survey results that enter the public domain with an adequate basis for judging the reliability and validity of the results. Website: www.britishpollingcouncil.org. Twitter: @BritPollingCncl
  • The first meeting of the inquiry is at The Royal Statistical Society, 12 Erroll St, London EC1Y 8LX at 1.30 pm on 19 June. Anyone who wishes to attend should register at BPC/MRS Polling Inquiry meetings
  • Members of the inquiry are Dr. Nick Baker, Group CEO, Quadrangle Research Group Ltd; Dr. Mario Callegaro, Senior Survey Research Scientist, Google UK ; Dr. Stephen Fisher, Associate Professor of Political Sociology, University of Oxford, who runs the Electionsetc website; Dr. Jouni Kuha, Associate Professor of Statistics, London School of Economics and lead statistician for the BBC/ITV/Sky exit poll; Prof. Jane Green, Professor of Political Science, University of Manchester and Co-Director of the 2015 British Election Study; Prof. Will Jennings, Professor of Political Science and Public Policy, University of Southampton, and a member of the Polling Observatory team; Dr Ben Lauderdale, Associate Professor in Research Methodology, London School of Economics and one of the team behind the electionforecast.co.uk website; Dr. Patten Smith, Research Director, Research Methods Centre, Ipsos MORI and Chair of the Social Research Association.
  • Details of Lord Foulkes’ bill can be found at: Regulation of Political Opinion Polling Bill [HL] 2015-16

For further information, please contact a member of the BPC Management Committee:

Simon Atkinson: 07791 680770
Nick Moon: 07770 564664
John Curtice 07710 348755

Details of Opinion Poll Inquiry Announced

Press Releases, ,

The British Polling Council (BPC) publishes today further details of the Inquiry into the performance of the opinion polls that it has established in collaboration with the Market Research Society (MRS).

​Under the chairmanship of Prof. Patrick Sturgis, Director of the National Centre for Research Methods at the University of Southampton, the Inquiry is charged with the task of establishing the degree of inaccuracy in the polls, the reasons for the inaccuracies it identifies, and whether the findings and conduct of the polls were adequately communicated to the general public. Due to report by 1 March next year, the Inquiry will seek and welcomes submissions from all interested parties, and is empowered both to make recommendations about the future practice of polling and, where appropriate, for changes in the rules of the BPC. The BPC and MRS are committed to publishing the Inquiry’s report in full.

​Eight people with professional expertise and experience in conducting and analyzing survey and polling data, have agreed to serve (unpaid) as members of the Inquiry. None of them were directly involved in conducting published polls during the election campaign. They are as follows:

  • ​Dr. Nick Baker, Managing Director, Quadrangle Research
  • ​Dr. Mario Callegaro, Senior Survey Research Scientist, Google UK
  • Dr. Stephen Fisher, Associate Professor of Political Sociology, University of Oxford, who runs the Electionsetc website
  • Dr. Jouni Kuha, Associate Professor of Statistics, London School of Economics and lead statistician for the BBC/ITV/Sky exit poll
  • ​Prof. Jane Green, Professor of Political Science, University of Manchester and Co-Director of the 2015 British Election Study
  • ​Prof. Will Jennings, Professor of Political Science and Public Policy, University of Southampton, and a member of the Polling Observatory team.
  • ​Dr Ben Lauderdale, Associate Professor in Research Methodology, London School of Economics and one of the team behind the election forecast website.
  • ​Dr. Patten Smith, Research Director, Research Methods Centre, Ipsos MORI and Chair of the Social Research Association.

​Information about the work of the Inquiry will be available via a website launched today at National Centre for Research Methods (NCRM). As a first step the Inquiry is inviting written submissions, which can be uploaded via the website. A public meeting will be held during the afternoon of 19 June at the Royal Statistical Society, London, where there will be an opportunity to discuss the work of the Inquiry. The event will be free to attend but registration will be required. Registration will open, via the NCRM website, on Tuesday 26 May. Further information and updates about the conduct of the inquiry will be made available on the website thereafter.

​Prof. John Curtice, President of the British Polling Council, said, ‘The polls clearly gave the public a misleading impression of the likely outcome of the 2015 election and this shaped the reporting of the campaign. The Council is committed to ensuring that there should a thorough and transparent investigation into what apparently went wrong, and how both the conduct and the reporting of the polls might be improved in future. We are deeply grateful to Prof. Sturgis and the members of the Inquiry, all of whom have substantial professional expertise in the methodology and analysis of surveys, for agreeing to conduct this Inquiry.’

Jane Frost CBE, MRS’ Chief Executive said, “As the world’s leading research association, we are actively supporting the British Polling Council in its investigation. We continue to support all of our accredited members in ensuring standards are met. Market research is a UK success story, the UK is a world leader in this sector, contributing over £3.6bn to the UK economy. We continue to learn, adapt and innovate.”

For further information:

Notes to Editors

  1. The full details of the Terms of the Reference of the Inquiry are appended to this release.
  2. The British Polling Council (BPC) is an association of polling organisations that publish polls. The Council promotes standards of disclosure that are designed to provide consumers of survey results that enter the public domain with an adequate basis for judging the reliability and validity of the results. Most of the companies that conducted polls of voting intention at the 2015 UK general election are members. Further details can be found at http://www.britishpollingcouncil.org/.
  3. The Market Research Society (MRS) is the world’s leading professional research association, training and regulating the research sector in the UK. The research sector is a major UK industry worth a conservative £3.6bn (GVA) per annum.
  4. The original announcement of the establishment of the Inquiry can be found at General Election: 7 May 2015

BPC/MRS Inquiry into the Performance of the Opinion Polls at the 2015 General Election.

Terms of Reference

  1. To assess the accuracy of the published opinion polls (both national and sub-national) at the 2015 general election.
  2. To evaluate whether any inaccuracies identified might be part of a pattern evident at previous elections.
  3. To investigate the causes of any inaccuracies that are identified. Potential causes to be considered will include (but not necessarily be limited to): the possible impact of late changes in vote preferences, sampling methods, interview mode, weighting and filtering, population coverage, item refusal, differential availability and willingness to participate, question order and wording.
  4. To assess whether the analysis or reporting of polls was influenced by a reluctance to be out of line with the published figures of other polls.
  5. To consult and seek relevant evidence from all appropriate stakeholders, including but not exclusively, polling organisations that are members of the BPC.
  6. To assess whether adequate information was provided and communicated to interested commentators and the public about how polls were conducted and what their results meant.
  7. To make, as it sees fit, recommendations for improving how opinion polls are conducted and published in future.
  8. To make recommendations, if necessary, for changing the rules and obligations of BPC membership.
  9. To submit a report to the BPC and MRS by 1 March 2016, with a view to its publication by BPC and MRS as soon as possible thereafter.