Sunday, June 5, 2011

Defending economic forecast models

I wanted to elaborate a bit on my reaction to Nate Silver's recent post on the limits of economic forecasts of elections. Silver's post is a good one. He explains one of the better forecasting models out there, Hibbs' bread-and-peace model, which explains something like 90 percent of the variance in elections just by examining real disposable income growth and troop fatalities in American military engagements. But then he notes that the model doesn't do a great job predicting out-of-sample elections. Silver produces this nice chart showing the difference between the model's forecasts and the actual presidential election results:
It's really not a shock that a model would do somewhat less well predicting out-of-sample results. After all, the model was based on only 10 elections -- if we add the data from more recent elections, we get a better model. And really, the model does just fine in post-sample elections. The one marginally large miss is 2000, for which the model over-estimated Gore's vote share by about five points. And just about all forecast models made similar errors with that election. We could chalk this up to poor campaigning skills by Gore, but really, should it be that much of a shock that the party whose president was impeached would underperform in the election a year later?

Where the model really falls down, though, is in the 1930s and 40s. There are some very good reasons why most economic forecast models of presidential elections begin with 1948:
  1. World War II was not really politics as usual in America. Criticism of the incumbent president was rather muted on the most central issue in American life for two election cycles, and troop deaths were atypically high in an atypically popular military engagement. 
  2. The elections of the 30s and 40s encompass the Great Depression, during which economic growth vacillated in fantastic and horrifying ways. Real per capita disposable income dropped by 24 percent in 1932! It grew by 26 percent in 1942! It turns out that voters' partisanship is somewhat robust to these shifts; even when Americans have lost a quarter of their income in a single year under Republican leadership, 40 percent of the country is still willing to vote Republican.  We shouldn't abandon a forecast model for falling down in such extreme circumstances.
  3. Prior to FDR's presidency, the federal government simply had a smaller role in influencing economic booms and busts. It wouldn't be shocking if voters had different expectations with regards to the government back then and voted accordingly.
I don't want to overly apologize for all errors in a model. These really are based on very few data points, and occasionally they'll get it wrong. But Silver makes a huge jump when he concludes, "It’s the economy, stupid. And everything else too." If you were to construct a forecast model of elections based on charisma, optimism, some the-candidate-you-want-to-have-a-beer-with variable, campaigning skills, or even unemployment, you would be catastrophically wrong a good deal of the time. You'd probably do no better than a basic coin-toss. Income growth, unpopular wars, and moderation/extremism really do affect votes on a large scale, while most other stuff really doesn't, despite the quantity of ink spilled over it.

1 comment:

Jordan Ragusa said...

Thank you for your thoughtful comments. I respect Nate for his work, but his post contained a number of straw man arguments (which is effortlessly destroyed). Josh had an excellent post that complements yours: