Stephen Fisher and Jonathan Jones

Tomorrow Americans go to the polls for the midterm elections. All 435 seats in the House of Representatives are up for election, as well as 36 Senate seats and a host of state and local offices.

The historic tendency for the President’s party to lose ground in midterm elections, is sufficiently strong to say that there is virtually no chance of the Democrats regaining control of the House. Meanwhile the experience of previous midterm elections with a second-term President suggests that the Republicans should have a decent of winning the extra six seats they need to take control of the Senate.

Not only history but also political and economic circumstances in recent months, and especially current polls for individual Senate races suggest the Republicans have, according to the main forecasters, at least a two-thirds chance of achieving a Senate majority.

We are not attempting to forecast the outcomes of any of these elections, but several others with excellent track records for US election forecasting are. This article purely provides some introduction and links to the forecasts and offers some commentary from a British election forecasting perspective. We consider the House forecasts before turning to those for the Senate.

US House Forecasts

As James Campbell notes, “Since the Civil War, the president’s party has gained House seats in only three midterm elections and has never gained more than 10 seats.” So it would be unprecedented for the Democrats to win the 17 extra seats they need to take control.

Even a cursory glance at the situation in the polls makes clear how safe the Republican position is. In 2012 they managed to win a majority of seats even though the Democrats won more votes. Since then approval of Obama has dropped and there has been about a 3 point swing the Republicans in the polls. More generally pretty much all the pointers suggest the Republicans should make gains.

Perhaps unsurprising then that the Washington Post Election Lab say there is a 99% chance of Republican control and Predictwise (which incorporate betting market data) even go so far as to say the probability is 99.9%.

While the evidence does seem overwhelming, these numbers are also so high that from a forecasting perspective they raise questions about the chances of a rare event. More importantly from a small-d democratic perspective, it seems unhealthy to have control of the House such a forgone conclusion.

Regarding the likely eventual seat tally, academic forecasts a couple of months ago suggested that the Democrats would lose between 4 and 16 seats. The average of the latest forecasts (here, here and three further ones summarised here) suggest the Republicans make a net gain of 8 seats.

Amazingly they all have very similar central forecasts: between 7 and 9 net Republican gains. This and the two-months out narrow range would be almost unimaginable for a set of British general election forecasts. In 2010 the final forecasts for the total number of Conservative seats ranged from 282 to 312 and the lack of precision is reflected in the average overall seats error which the ranged from 26 at best to over 50.

But again, the more important issue is the lack of change that is being predicted despite a swing of about 3 points in the (generic ballot) polls. On average for the post-war period in Britain the two main parties have gained or lost 15 seats for every percentage point swing. Allowing for difference in size this means we would be expecting to see more like 30 seats changing hands this week if the House of Representatives were as responsive to changes in national party support as the House of Commons.

While we could quibble about the desirable level of responsiveness, electoral system bias is a plague on all first-past-the-post houses most of the time. It is currently worse in Britain than the US. But, by overtaking the Democrats in the share of the vote and simultaneously securing incumbency advantage in a few more seats, Republicans are likely to strengthen the electoral system bias towards them.

US Senate Forecasts

The Republicans need to win six more seats to control the Senate. James Campbell notes that, “in the 25 midterm elections conducted since the popular election of senators in 1912, the out-party has (surprisingly) gained a net of six or more Senate seats about half of the time (12 of the 25). Focusing on more recent midterm elections, the president’s party has lost six or more seats in four of the seven midterms from 1986 to 2010.” Moreover, “Since 1912, of the 14 midterms in a party’s second presidential term or more, the out-party has gained six seats or more nearly two-thirds of the time (9 of 14).”

So, history alone suggests the Republicans should have a good chance of achieving the goal. Add to this that the Senate seats being fought this year were last contested in the heady days of Obama’s first election 2008 when Democrats achieved 8 gains and a 7-point lead in the Senate vote. Even with new incumbency bonuses, victories off presidential coattails like these were always going to be hard to defend.

The statistical (and so close to the election almost entirely poll data driven) forecasters all now agree that the Republicans are more likely than not to take control. But while four of them estimate the probability to be close to 73%, Sam Wang’s ‘Princeton Election Consortium’ puts it at 60% while Washington Post’s Election Lab says 94%. This latter figure seems too high, but see their discussion here.

It is less clear whether Sam Wang is being too cautious. There was a ‘nerd fight’ between Wang and Nate Silver over modelling uncertainty. There is a good basic explanation of some of the issues by Nate Silver here, his more direct critique of Wang’s model here, and Wang replies here. Vox gives a more impartial description of the disagreement here.

The irony is that that debate was weeks ago when Sam Wang’s forecast was 75% for Democrat retention while others more ambivalent. Now Wang looks more ambivalent and the others more confident. But the underlying issues are the same. Wang’s overall uncertainty masks generally greater certainty over particular races but a lower point estimate for the total number of Republican seats. Whereas other statistical forecasters using state polls have central forecasts of 52 or 53 Republican seats, Wang’s is 51.

But as with the House forecasts, these are all remarkably similar, especially from a British perspective.

As well as the statistical forecasters, we should note that the Cook and Rothenberg Political Reports and Larry Sabato of the University of Virginia give more qualitative descriptions of the state of play for each race. It is striking that these three forecasters seem to suggest more uncertainty over more seats than most of the statistical ones and so an overall impression somewhat closer to that of Sam Wang but for different reasons.

Which Senate seats will change hands?

The New York Times has a helpful overall summary for state-by-state detail of the forecasts here. The most striking feature is that all the statistical forecasters are saying broadly similar things.

The table shows us where six gains from the Republicans are most likely to come from.

As testimony to the power of incumbency advantage, of the five states where the sitting Democrat is retiring, only one (Michigan) looks sure to be held while three are practically certain Republican gains (Montana, West Virginia and South Dakota). Meanwhile Iowa looks likely to go the same way. On the other side, Georgia is one of only two Republican seats in jeopardy, and it is no accident that it is one of the three cases of Republican retirement.

Of the Democrat seat with incumbents standing for re-election, Arkansas looks highly likely like to go. Then the Republicans need just two of Colorado, Louisiana, Iowa, and Alaska (where the probabilities are in their favour) while North Carolina and New Hampshire are longer shots. They may need more than two if they lose Georgia.

The only sitting Republican senator who is at danger of losing is Pat Roberts of Kansas. He is fighting a tight race against an independent Greg Orman. If that were not unusual enough, Orman has said that he would caucus with whichever party has a majority, but if the Republicans end up with 50 seats to the Democrats’ 49, he would see which would commit to his positions on issues. Because Orman’s position on most issues is closer to the Democrats than the Republicans, Nate Silver handles this by giving him a 75% chance of caucusing with the Democrats in that tie-breaker situation.

But, it may well take some time to discover which party Orman will join, not least because there’s a good chance we won’t know which party controls the Senate on Wednesday.

Both Louisiana and Georgia require a candidate to get more than 50% of the vote to be elected – if no one does, the top two go through to a runoff. It’s very likely Louisiana’s Democratic Senator Mary Landrieu will have to face Republican Bill Cassidy again on 6th December, and reasonably likely that neither David Perdue nor Michelle Nunn will get over the line in Georgia, forcing a second round that won’t be held until 6th January.

Meanwhile the closeness of races in Alaska, Kansas and elsewhere mean we could well see long counts or even recounts before a winner is declared (particularly in Alaska, where polls don’t even close until 5 o’clock on Wednesday morning on UK time).

Nate Silver’s model suggests there’s a roughly 60% chance that one or more of these factors will mean we’ll have to wait longer than usual to know who has a majority.

But back to the forecasts…

Statistical forecasts have a strong track record. In 2012 Drew Linzer, Simon Jackman, Nate Silver and Sam Wang each correctly predicted all or all but one of states (and DC) results for the presidential election. Technically their work is very impressive.

The statistical analyses based forecasts are heavily if not almost entirely driven by opinion polls this close to the election. Also throughout the campaign the absence of a presidential contest weakens the correlation between national and state level trends in party support and so proper state-level polls are vital. In contrast with Britain, where constituency-level polls are still relatively few, state-level polls are many and have a relatively good track record. There are good reasons to think that they will do a good job again this year.

However there are also good reasons for concern. Nate Cohn argues that because of greater difficulties in reaching them the polls tend to undercount the Democrats. HuffPost Pollster’s Mark Blumenthal has a good discussion of the historical bias in the polls. Apart from 2002 when there was a big swing to the Democrats, the polls have overestimated the Republicans in midterm Senate polls since 1998, albeit of varying magnitude. More recently he discusses various ways in which the polls might be wrong this time and concludes that, “A similar systematic misfire in 2014 could reverse Republican leads in a small handful of states, but it would require very close margins on the final polling averages. Despite a fairly consistent understatement of Democrats in 2006, 2010 and 2012, the polling averages HuffPollster examined erred in forecasting the ultimate winner in just two of 25 contests.”

Nate Silver has argued that the polls are just as likely to be biased towards the Democrats as to the Republicans: “On average since 1990, the average bias has been just 0.4 percentage points (in the direction of Republicans), and the median bias has been exactly zero.” So his model allows for bias in either direction.

Silver’s more recent analysis striking argues that state level polling bias is negatively correlated with prior partisanship (as measured by the Bush-Kerry vote). There are no Democrat battleground states; the polls ought to be relatively unbiased in the purple states (e.g. Colorado and Iowa); but they should be biased against Republicans in traditionally Republican states like Kansas and Alaska.

Since there are good reasons to believe the bias risk could go either way and that has been factored in to some probability estimates, we are left with little reason to doubt the median estimate of 73% chance of the Republicans taking control. But given the risk of extreme events not reflected in the historical data, it might be safer to say that the chances were between two thirds and three quarters.

While Democrat retention of the Senate would be mildly awkward for most of the statistical forecasters it would only be a severe blow for the Washington Post model who have put the chances at just 4%.

But there are also some significant differences between the nature of the forecasts at the state level which will allow for some interesting post-mortem analysis. Also, although this article has focused more on the current forecasts, it is more important to evaluate their trajectories. For instance, while Sam Wang’s polls-only model has moved from predicting Democrat retention at the start of September, Nate Silver’s polls-and-fundamentals model has consistently pointed to Republican control as the most likely option.

Concluding thoughts

Even if the Republicans end up controlling both houses of Congress for 2015 and 2016 this does not mean that they will be able to pass whatever legislation they like. The Democrats are certain to retain the 41 seats they need to filibuster in the Senate, and the Republicans are very unlikely to win the 290 seats in the House they would need to overturn a Presidential veto. Also, congressmen are often more loyal to their constituents than their parties, meaning that party majorities do not always win the day. There is a good discussion here of what Republican control of the Senate might mean politically.

Appendix: Notes on some of the key races

Kansas: One of the most crucial races in determining whether the Democrats retain control of the Senate doesn’t actually feature a Democratic candidate. The party’s nominee, Chad Taylor, withdrew from the race in early September to give independent Greg Orman a clear run at Republican incumbent Pat Roberts. The polls are very close and the unusual nature of the contest makes the outcome particularly uncertain. The forecasters either make it a tossup or slightly favour Orman.

Georgia: The open seat left by retiring Republican Senator Saxby Chambliss is being hotly contested by Republican nominee David Perdue and Democrat Michelle Nunn (whose father Sam held the seat from 1973 to 1997). The polls – and forecasters – slightly favour Perdue to hold this seat for the GOP.

Iowa: Democrat Tom Harkin is retiring, and Republican Joni Ernst is slight favourite to pick up the seat for the GOP, with a small lead in the polling averages over Democratic nominee Bruce Braley.

Alaska: In 2008, Mark Begich defeated Republican incumbent Ted Stevens – who had just been convicted of corruption – to become Alaska’s first Democrat in Congress since 1981. He’s facing a tough fight to win re-election, though, with the polls tending to show Republican Dan Sullivan slightly ahead. That, coupled with Alaska’s red-state nature, is enough for the forecasters to make the Republicans favourites to retake the seat, though — as Harry Enten and Nate Silver point out – polls in Alaska have tended to be less accurate than those in other states, adding to the uncertainty here.

North Carolina: The polling is very tight (three of the last seven show the race tied), but Democrat Kay Hagan is slight favourite to hold onto her seat against Republican Thom Tillis. Some forecasters (HuffPo, Sam Wang) have the race at almost evens, while others (Preictwise, Washington Post) see Hagan as a roughly 1-4 favourite.

Colorado: The forecasters had first-term Democrat incumbent Mark Udall as slight favourite back in September, but the polls have since moved to his Republican opponent, Cory Gardner, who now leads by around two points in the averages. Gardner’s now the favourite to win, but HuffPost Pollster still give Udall a 38% of keeping his seat, whereas the Washington Post only give him a 3% chance.

Louisiana: Bill Cassidy has had a pretty steady four- or five-point lead over Democrat incumbent Mary Landrieu in the polling average for more than a month now, and all the statistical forecasters make him a clear (if not decisive) favourite to gain the seat for the Republicans. The three non-statistical forecasters (Cook, Rothenberg and Sabato), though, have the race as a tossup.

New Hampshire: Another Democrat 2008 winner seeking a second term, Jeanne Shaheen looks to be in better shape than Begich, Hagan or Udall. Her Republican challenger is Scott Brown who represented Massachusetts in the Senate after winning the special election following the death of Ted Kennedy, before losing to Elizabeth Warren in 2012. Brown has closed up in the polls in recent weeks, but still trails Shaheen by two or three points in the averages. The forecasters make Shaheen the favourite by anything from 60-40 (HuffPo) to 99% (Washington Post).

Kentucky: Senate Minority Leader Mitch McConnell would become Majority Leader if the Republicans take control – but only if he keeps his seat against fierce opposition from Democrat Alison Lundergan Grimes. Grimes had looked very competitive, pulling level with McConnell at times earlier in the year, but the recent polling has shown reasonably consistent leads for McConnell. He now leads by four or five in the averages, and the forecasters now make him a heavy favourite.

Arkansas: Republican Tom Cotton leads incumbent Mark Pryor by around 5 points in the polling averages, making this a likely GOP gain. The forecasters disagree over how likely though, with HuffPost Pollster giving Cotton a 74% chance of winning while Predictwise and the Washington Post make his victory a near-certainty at 99% plus.