Monthly Archives: May 2018

British Polling Council Introduces New Rule on Uncertainty Attached to Polls

Press Releases,

The British Polling Council announces today that it has introduced a new requirement on its members when reporting estimates of vote intention. This requirement is an obligation to publish a statement of the level of uncertainty that has historically been associated with polls of voting intention.

The statement reads as follows:

‘All polls are subject to a wide range of potential sources of error. On the basis of the historical record of the polls at recent general elections, there is a 9 in 10 chance that the true value of a party’s support lies within 4 points of the estimates provided by this poll, and a 2 in 3 chance that they lie within 2 points.’

The statement has been developed following an analysis of the performance of the final polls at every general election since 2001. That analysis revealed that 90% of the estimates of a party’s share of the vote in the final polls were within four points of the eventual true result, while two-thirds were within two points.

Hitherto, it has often been noted that, according to statistical theory, there is a 95% chance that in a poll of a thousand people the true value of a party’s share of the vote lies within three points of the poll’s estimate. However, this statement refers solely to the error that may be occasioned by the chance variation to which all sample surveys are subject. It does not take into account any of the other many possible sources of uncertainty in a poll. The new BPC statement attempts to reflect all of the sources of uncertainty that practical experience suggests is associated with polling in Britain.

The statement is intended to discourage the media and others from reporting small changes in a party’s support – or variation between pollsters – as firm evidence of a significant change in its popularity.

The new BPC rule has been introduced in response to recommendation 11 of the Inquiry into the 2015 British general election polls that was chaired by Prof. Patrick Sturgis. This recommendation read that:

‘BPC members should provide confidence (or credible) intervals for each separately listed party in their headline share of the vote’.

The introduction of this requirement means the BPC has now responded to all of the recommendations that were made to it by Prof. Sturgis and his colleagues.

Notes to Editors

  1. The British Polling Council (BPC) is an association of polling organisations that publish polls. The objectives of the Council are to ensure standards of disclosure that provide consumers of survey results that enter the public domain with an adequate basis for judging the reliability and validity of the results. Through full disclosure the Council aims to encourage the highest professional standards in public opinion polling and to advance the understanding, among politicians, the media and general public, of how polls are conducted and how to interpret poll results.
  2. For further details of the Council see http://www.britishpollingcouncil.org/
  3. The report of the Inquiry into the Performance of the Polls in the 2015 Election is available at the National Centre for Research Methods (NCRM) pdf

Minutes of the Annual General Meeting 2018

AGM Minutes,

The Meeting was held at ORB on 20th February 2018

Present

Sir John Curtice, President
Nick Moon, Secretary/Treasurer
Simon Atkinson, Management Committee
Johnny Heald, ORB
Ivor Knox, Panelbase
Anthony Wells, YouGov
Gideon Skinner, Ipsos MORI
Laurence Stellings, Populus
Jansev Jemal, ComRes
Adam Drummond, Opinium
Luke Taylor, Kantar Public
Damian Lyons Lowe, Survation

1. Apologies for absence

BMG, Lucid Talk, and ICM

2. Minutes of the last AGM

The minutes were approved

3.Officers’ reports

  • Financial report

    We currently have £9,985 in the bank, and £3,750 of outstanding membership fees (from 2017 and 2018), making a due total of £13,735. There are expenses expected, but probably no more than £500.

    NM raised the possibility of another fees holiday but it was agreed that the BPC needed to build a reserves fund in case the need for another enquiry, and fees would be levied as usual in 2019. With the existing members NM reported that annual subscription income is £2,750.

    It was agreed NM would send a note of thanks to MRS thanking them for their financial support for the Inquiry.

  • Disclosure issues dealt with by the officers

    Nothing that fell within our remit. ComRes audience recruitment raised a new serious issue about whether this fell within the Rules. The Officers decided it didn’t, and a clarification was published on the BPC website making this clear

  • Membership applications dealt with by the officers

    DCYFOR have become members

4. Appointment of the officers

  • President
    NM nominated JC, SA seconded, elected nem con
  • Secretary
    JH nominated NM, DLL seconded, elected nem con
  • Management Committee member
    AW nominated SA, LS seconded, elected nem con

5. President’s report

The report raised the issue of whether we need to think about rules to cover Multilevel Regression and Post-stratification (MRP). It was agreed we keep a watching eye on this for now. All other main points from the report are covered in later agenda items.

6. House of Lords Committee

JC and SA attended on behalf of BPC, NM and various BPC members also gave evidence.

BPC might have to make some kind of holding statement quickly after the final statement is released. We have already said we’re opposed to both statutory regulation and banning polls. We have slightly moved into quality domain by having enquiries into elections and agreeing certain procedures as a result, so may be under pressure to venture further.

JC encouraged members who were not CEOs of their organisation to ensure that their CEO was fully aware of the potential difficulties following publication.

Cttee due to report by end of March.

Agreed that JC should be authorised to give a strong defence of the value of self-regulation – citing all the openness and attempts to improve that happened after the election. We are much more transparent than most other countries.

It was agreed that the BPC would hold at least one mid-Parliament meeting, ideally within the Palace of Westminster, as well as the pre and post-election sessions to explain latest thinking.

7. Reporting uncertainty in the polls

Gideon introduced paper produced by Luke Taylor and him. Aim was to try to find a simple approach of explaining likely levels of uncertainty in polls. They decided best way, to avoid having to explain statistical theory to lay people, was to use history of reliability of polls in previous elections. Looked at all final polls done by BPC members (including 2001 before BPC existed).

Suggested level of uncertainty of +/-4pp, (achieved by >90% of polls), but also mention that most polls are within +/-2pp.

The Sturgis bootstrapping approach, achieved by taking repeated random sub-samples from polls, was felt to be harder to understand. Also bootstrapping ignores bias rather than variance by not comparing with actual result
Members would be free to calculate and publish their own.

Agreed to principle of what the paper says.

We need a PR statement to accompany the change in rules.

8. Proposed rule change following item 7

Insert at the end of the bullet points at Rule 2.4.

In the case of a poll of voting intentions for an election, include in the published documentation the following statement: ‘All polls are subject to a wide range of potential sources of error. On the basis of the historical record of the polls at recent general elections, there is a 9 in 10 chance that the true value of a party’s support lies within 4 points of the estimates provided by this poll, and a 2 in 3 chance that they lie within 2 points.’

This amendment to take effect from 1 April 2018.

GS asked all members for comments, and to check their own companies’ figures.

JC proposed NM seconded nem con.

9. Any Other Business

SA suggested we look again at FAQs and journalists’ guide to opinion polls on the website in the light of the Lords Cttee

JC suggested we investigate cost of updating the design of the website.