Stephen Fisher, 11^th September

The vote intention polls for the Scottish independence referendum seem to have been taken largely at face value by commentators, politicians and even the financial markets. In particular, the roughly equal split of the vote between Yes and No in several recent polls is being interpreted as if it implies that the result of the referendum is likely to be close. But how accurate are opinion polls as predictors of referendum outcomes and how accurate are polls for elections in Scotland more generally?

Looking at 16 recent and/or pertinent constitutional referendums (listed below) we can consider the general accuracy of headline figures from final polls setting aside Don’t Knows.

In no less than 12 out of the 16 cases was the vote for Yes (the change option) in the final polls was higher than that in the eventual outcome. The twelve include the referendums on the Alternative Vote in 2011 (8 point difference), Welsh devolution in 2011 (4 point difference) and 1997 (3 points), the Good Friday agreement in 1998 (3 points), Quebec independence in 1995 (4 points) and Scottish devolution in 1979 (3 points). The 12 also includes the 4 point over estimation of the Yes vote in the 1975 European Community referendum, which was strictly speaking a vote for the status quo, but only a recently established one.

Strikingly one of the remaining four cases is the Scottish Parliament referendum in 1997, which involved two questions. The final polls (excluding Don’t Knows) underestimated the votes for establishing a parliament and for giving it tax raising powers by 1 and 4 percentage points respectively.

But it is also notable that, together with the Irish referendum in May 2012, the 1997 Scottish referendums were the only ones where the polls seemed to underestimate the vote for change by more than a percentage point. Moreover, both were comfortable Yes victories of with over 60% of the vote. While there are examples of close referendums in which the Yes side did worse than expected from the polls (such as Quebec in 1995 where the final polls said Yes but the result was No) there does not seem to be a precedent for a close referendum at which final polls overestimated the Yes vote.

The tendency for final polls to differ from the actual result does not necessarily mean that referendum polls are biased towards Yes responses. It might be that the Don’t Know responses split disproportionately towards No, or those in favour of the proposition are less likely to turnout to vote, and late swing is a possibility. Whatever the reason, the experience of referendum polls in the UK and internationally suggests that final polls without Don’t Know responses are typically flattering for the Yes camp.

The record of polls in recent elections in Scotland is should also be considered sobering for the Yes camp. True, the SNP secured a majority of seats in the Scottish Parliament in 2011 well exceeding the expectations from the pre-election polls. But they did so on just 45% of the vote, and the SNP have done considerably less well in the local and Euro elections since then. Indeed, the SNP managed only 29% of the vote in May this year, well below the 37% the final polls suggested they would get. They similarly under performed relative to the polls at the 2010 general election. So election results provide mixed evidence, but more reason to believe that the nationalist cause is being over rather than under represented in the referendum polls.

In addition to analysing polls at previous referendums and elections it is important to also consider how the particular nature of this referendum might lead to a discrepancy between the final polls and the actual result.

One possibility is differential turnout, with one side’s supporters more likely to vote than another. At elections some pollsters try to adjust for this by weighting respondents according to their self-reported likelihood of voting in their final polls, but this does not always make the headline figures better. The last YouGov referendum poll had 91% saying that they would certainly vote, leaving little room for weighting to make much difference.

It is speculative to suggest that it will be psychologically more compelling for supporters of independence to go to the polls than it will be for others to vote to defend the union. But we do know that the grassroots campaign activity of the Yes campaign is more intense than that of the No campaign, which might both lead to more persuasion between now and the vote but also greater Yes turnout on the day. That said, if turnout is as high as it is expected to be, the scope for significant differences between the camps in turnout rates is relatively limited.

Another issue is how the Don’t Knows will split. Research from Canadian electoral reform referendums suggests that Don’t Knows split disproportionately towards the status quo, and many commentators think that that is what will happen next week in Scotland. This does seem more likely than a split in the other direction since, but a big movement from Don’t Know to No between the final polls and the vote seems unlikely. If someone is still undecided at the end of a long and intense campaign on such an important issue it probably means they really are not sure which way they will go.

A third consideration is whether the respondents in the polls are a representative sample. John Curtice has an excellent piece on this which argues that although previous non-voters are less likely to appear in polls they are actually also less likely to vote Yes. So there is not much in the poll data to suggest that there is a strong pro-Yes tendency amongst the kinds of people that pollsters have been unable to reach.

Finally, there is misreporting. The pro-Labour bias in the 1992 opinion polls was attributed in part to a ‘spiral of silence’ or ‘shy Tory’ effect and in part to late swing. According to some, people were less likely to report favouring an unfashionable option (spiral of silence) or flirted with the idea of the more hopeful but risky option only to get cold feet at the last moment (late swing). Applying these notions to the Scottish referendum suggests the possibility of a Yes bias in the final polls.

So overall the evidence is mixed, but not balanced. It seems more likely that the headline poll figures excluding Don’t Knows are over rather than under-estimating the vote for Scottish independence, and this will be especially true of the final polls.

Acknowledgements: Thanks to Alan Renwick for much of the data and helpful comments on an earlier draft.

Notes on the data: The referendums discussed above were Austria (2013), Ireland (2008, 2009, 2011, May 2012, Abolition of the Seanad 2013), New Zealand (2011), Northern Ireland (1998), Quebec (1995), Scotland (1979 and 1997), Sweden (2003) UK (1975, 2011), Wales (1997 and 2011). There are others that I would like to add but I haven’t tracked down the data yet. Any contributions much appreciated.