Stephen Fisher and Rosalind Shorrocks

There is a lot of evidence from the academic research on forecasting that suggests it is a good idea to combine information from different sources (e.g. here). In US and German elections generating a forecast by combining the forecasts of others has a good track record. For the upcoming US presidential election pollyvote.org provides an average of different forecasts together with excellent summaries and discussions of the different methods and forecasts.

We have developed a similar method of combining forecasts for the UK’s referendum on EU membership on 23rd June 2016. The summaries of the average forecast win-probability and share of the vote for Remain by method and then overall are presented in the tables below. Note that there are different components for each because some of the source forecasts provide only probabilities or only vote share.

Share of the vote	Remain	Leave
Combined forecast (mean)	53.6	46.4
Betting markets	53.9	46.1
Polls	51.4	48.6
Expert forecasts	56.0	44.0
Volunteer forecasts	54.6	45.5
Poll based models	54.0	46.0
Non-poll based models	52.0	48.0

	Probability that Remain wins
Combined forecast (mean)	66.8
Citizen forecasts	64.3
Volunteer forecasts	71.2
Prediction markets	65.4
Betting markets	62.2
Polls	63.0
Poll based models	74.7

Most strikingly all the forecasts point to a Remain win. There is also a remarkable degree of consensus across very different methods, especially given that some are suggesting considerable uncertainty.

These averages and combined forecasts were collected from the sources on the 19^th April. We will update these forecasts from time to time.

Details of methods and sources are below. We do not endorse any of the component forecasts. As mentioned below, there are a couple of prediction markets with very low trading activity that we have not used. Otherwise these are all the useable forecasts that we know about. We have not excluded any forecasts based on our judgement of quality. Please do let us know if there are any other published forecasts that you think we may have missed and could be using. More generally, the methodology is still under development so comments very welcome.

Citizen forecasting

Citizen forecasts come from the results of representative surveys of voters, asking them what they think the outcome will be. Such voter expectation surveys have an excellent track record, arguably better than polls, prediction markets, quantitative models or expert judgment for US presidential elections. The percentage who think that Remain will win is taken as a collective estimate for the probability of a Remain win. Results of voter expectation surveys are listed here. For each pollster we take the average of the last two such polls within the last three months, and then average across pollsters.

Expert forecasts

At the moment in this category we just have the results of the Times Red Box sweepstake podcast contributors as our experts, plus some others we can identify as experts.

We considered taking the proportion of experts who gave predicted Remain shares above 50% as a pseudo probability of a Remain vote. However this figure would have been 95% and not fairly represent the uncertainty felt by many of the expert contributors. Nonetheless this unity here is striking. Moreover, there are various other published expert forecasts but they are not used here because they do not provide figures for either the share of the vote or the probability of one side winning. They too overwhelmingly point to Remain winning.

We hope that there will be more systematic surveys of academics, pollsters and journalists in due course.

Volunteer forecasts

Philip Tetlock’s Good Judgement project encourages people to forecast the outcomes of various social and political events and helps them learn and improve their forecasting skills. Tetlock claims that given some training, effort and practice, reasonably intelligent citizens can forecast better than experts, at least collectively. One of their forecasting challenges is the outcome of the Brexit referendum (here).

In addition we use the volunteered contributions to the Times Red Box sweepstake, i.e. the participants excluding the podcast contributors. For some academics and others that we know of we have reclassified their forecasts as expert. The proportion of these volunteered contributions giving predicted Remain shares above 50% is again taken to be the probability of a Remain vote. The median Remain vote share is used for the share forecast.

Prediction markets

These are websites which allow people to bet on the outcome directly with other participants, without a bookmaker setting the odds (explanation here). They are much lauded as a forecasting tool by many economists and business people because they draw on views from a wide range of people willing to risk their own money. For election forecasting they arguably have a better track record than polls, quantitative models and expert judgement.

For the Brexit referendum the prediction market websites only have markets for which side will win not the share of the vote. We use data from predictit and hypermind. We also use spread-betting markets from sportingindex and ig, taking the mid-point of the spread as the predicted probability or vote share.

Because of low trading rates we do not use ipredict.

Betting markets

These are traditional bookmakers. Even though the odds are formally set by the bookmakers, with enough people betting they are primarily driven by what the punters are willing to accept. We average across major bookmakers listed here after correcting for the over-round (whereby the sum of the implied probabilities from the published odds is more than 100).

Bookies allow people to bet on the share of the vote within particular bands (e.g. 45-50% Remain). To generate a combined vote share forecast we take the mid-points of the bands and weight them by the (corrected) implied probabilities. For large bands that extend to 0 or 100 we do not use the mid-points but figures five points from the interior bound. For example, if the band is 75% to 100%, we use 80% for the share calculation. Implied probabilities for these extreme bands are very small so the choice of mid-point makes little difference to the calculations.

Polls

For the share of the vote we use the average of the polling averages that are published by whatukthinks, Ben Stanley, Number Cruncher Politics, the FT, and ElectionsEtc. These polling averages typically aim to correct for differences between pollsters and so they should not fluctuate too much according to whether the most recent polls were online or telephone or from a particular company. So what we are calculating is a poll of polls of polls!

Since polling averages do not reflect the range of variance in the polls very well, we generate a pseudo probability for Remain winning from the proportion of polls that have Remain ahead. For this we take just the last two polls from each company-method combination within the last two months. So if a company has published two online polls and two phone polls in the last two months these are treated separately. Here we make no attempt to balance between online and telephone polls even though there are slightly fewer companies doing phone polls and they more clearly point more towards Remain.

Poll based forecasting models

Polls are a snap shot of opinion at the time they are taken. The historical relationship between polls and referendum outcomes tells us something about the direction and extent of any likely change in opinion, as well as the level of uncertainty we can expect of the outcome. Both ElectionsEtc.com and Number Cruncher Politics provide forecasts of this kind, but the latter provides only probabilities without a vote share forecast.

Non-poll based models

A traditional approach to election forecasting is to use historical data to develop a statistical model based on factors that are expected to influence the vote. These factors are often referred to as the “fundamentals”. See here for an introduction to these kinds of models for US elections.

The only such published model for the Brexit referendum that we know of is from Matt Qvortrup (see here and here). His latest estimate is here.