By Stephen Fisher and Alan Renwick
The UK will have a referendum on whether to remain in or leave the EU on 23rd June 2016. We have developed a method for forecasting the outcome based on current vote intention polls and analysis of opinion polls from previous referendums from the UK and across the world.
In the run-up to the 2014 Scottish Independence referendum we wrote about how across various referendums there has been a tendency for the eventual vote for change to be lower than it was in polls over one-month out from referendum day, and for even final polls to overstate support for change. But these observations were based on relatively small samples of referendums.
Now, as part of a broader academic project, we are analysing over 1,600 polls from 283 referendums in 41 democracies. For the purposes of forecasting the Brexit referendum outcome, we have used just referendums in the UK or on the EU. This still includes 848 polls from 45 referendums, and for the model specification we draw on the lessons from our analysis of the broader data set.
Despite the addition of many more referendums from many more countries, the patterns we previously found largely hold up. Support for change tends to decline as referendum day approaches, but not so much for referendums on the EU as opposed to domestic political reform. Even final opinion polls tend to show higher support for change compared with the eventual outcome. Most importantly, there is a lot of variation across referendums in the extent of these tendencies, so much so that there are plenty of cases that go in the opposite direction. The experience of previous referendums suggests that the eventual Remain vote is likely to be higher than that in polls currently, but not by much – just less than 3 percentage points – and the gap between the current polls and the result is only somewhat more likely than not to be positive for Remain rather than negative.
Previous referendums are perhaps more valuable in telling us by how much the eventual outcome might differ from current polls than for predicting the direction of change. Our forecast share of the vote for Remain has a big margin of error of + or – 14 percentage points.
A critical part of our forecasting method is how we combine recent opinion polls on referendum voting intention. There is a remarkably consistent 9 percentage point difference between online and telephone polls in the level of support for Remain. This is excluding Don’t Knows, who are much more prevalent in online polls, where Remain voters are correspondingly fewer. The reasons for this effect of the mode of interviewing are not clear. Sampling, prompting, interviewer effects and question wording may all play a role. We will continue to analyse this situation but for the moment we are generating a polling average based on the last six polls but benchmarked to the mid-point of online and telephone polls. In effect this means adding 4.4 to the Remain share in online polls and subtracting the same amount for telephone polls. The six most recent polls, as of 11th March 2016, were all conducted online, and showed an average of 51% for Remain, excluding Don’t Knows. After adjusting for the mode effect, our polling average is 55% Remain.
Combining that with the direction of change suggested by previous referendums produces a forecast share of the vote for Remain of 58%.
The + or – 14 point margin of error means that the eventual Remain vote could reasonably be expected to be anywhere between 44% and 72%.
Values closer to 58% are more likely, but there is still a reasonable chance Leave could win. Our estimated probability of a Leave win is 13% and so Remain have an 87% chance of winning.
Analysis of the historical data and development of the forecasting methodology remains a work in progress, so the character of the forecast might change in future. We are painfully aware that there are many valid considerations about the particularities of this referendum that our forecasting method does not reflect, and also there are reasonable doubts about how comparable, and so useful, the experience of polling at any other previous referendum is for this one. Nonetheless, we feel that the method is reasonable enough to make this first forecast.
Thanks to Jack Sheldon, Roberta Damiani and Johnny Runge for data gathering. We will write more about both our analyses of public opinion in the run-up to referendums and our forecasting method in due course.