Saturday, November 12, 2011

Economic forecast models: the pushback

Nate Silver's 2012 prediction model has produced some interesting responses across the blogosphere. Brendan Nyhan notes that Silver's estimate for the effects of candidate ideology in presidential elections just has to be too big*, and he finds that Silver's model underperforms when compared to others. John Sides responds to Mike Tomasky's dissing of political scientists. Then Alan Abramowitz offers a model that includes an important variable that measures how long the incumbent party has held the White House, and Sean Trende pushes back a bit on that one and on the whole concept of economic forecasting.

I want to respond to what Trende wrote, but I'd first like to point out that it's fantastic that we're even having this debate. Just a few years ago, it seemed like political scientists were tearing their hair out just trying to convince political journalists that elections weren't determined entirely by campaign activities, and that the economy and other fundamental aspects of the political environment might be relevant. Now, the debate among political scientists and journalists appears to be more like, "Come on, the economy doesn't explain everything." So I feel like this discussion has moved in a very good direction in the past few years.

Okay, back to Trende's piece. He ticks off a bunch of things that you have to believe if you're going to accept the validity of an economic forecast model. For example:
First, at a basic level, you have to accept that something as complex as voting can be reduced to a simple, three-variable equation. And you have to accept that this equation is linear.
Well, no, you really don't. Now, we have good evidence that you can explain a very high percentage of what goes on in elections with just two or three variables, but that doesn't mean that everyone's vote choice is a result of just those variables. These models do have error terms. Sometimes other things can affect votes. They just usually don't.

And just because most of these models are linear, that doesn't mean that they have to be. If you get way out in the tails of economic performance, you can see that the effect on votes isn't quite linear. Hoover did better in 1932, and FDR did worse in 1936, than a linear model of economic growth would predict, if for no other reason than that any major party presidential nominee is guaranteed close to 40% of the vote; their hardcore partisan supporters simply won't defect no matter how bad things get. But most elections don't occur under such extreme conditions, and the linear model works quite well for those.

Here's another point Trende makes:
You have to accept that there is no problem predicting the president’s vote share from only 16 data points.
That's silly. Of course there's a problem with that. But that's all the cases we have, and they work pretty well. Indeed, it's pretty amazing we get such robust results from so few cases. And keep in mind that a lot of the truths we cling to in American politics, such as "the president's party loses House seats in midterm elections" or "Democrats lose when they nominate liberal New Englanders," are based on this many cases or fewer.
You have to accept that presidential elections haven’t changed at all over the past 64 years. ... You have to accept that the enfranchisement of African-Americans and poor whites in the South, as well as the enfranchisement of 18-to-21-year-olds nationally, had no effect on the outcome of the later races. A casual glance at the results of the 2008 elections would seem to suggest otherwise.
No one I know is claiming that elections are exactly as they were 64 years ago. But the same basic trends do seem to hold: voters blame the incumbent party when the economy underperforms and reward them when the economy does well, and they tend to turn against parties that have been in power a long time. I don't know why he singles out 2008 as some sort of evidence that these fundamentals have changed. In fact, such forecasts nailed the '08 results within a single percentage point.
You have to accept that anything that happens past the end of the second quarter of an election year matters only at the margin. If the economy absolutely collapses, and a previously popular president goes into Election Day with a 20 percent approval rating amid a full-scale depression, where the economy is contracting by 10 percent a quarter, it wouldn’t matter much. If we are attacked and enter a war, it wouldn’t matter much. If a president becomes mired in scandal, it wouldn’t matter much.
Again, no one argues this. A lot of modelers pick the second quarter as a cut-off because it allows time to make a forecast several months before the election while still capturing much of the economic activity on which the incumbent party will be evaluated. It's very rare that an economy that's humming along at three percent growth for a year or more will suddenly plunge into recession the quarter prior to an election. That certainly can happen (and it kind of did in 2008), but it's a very rare event. Similarly, if the forecast models showed Obama likely to win next year, but he decided to shoot Tom Hanks in the face on live TV on October 31st, yeah, he'd probably lose the election. It's not that last-minute twists don't happen, it's just that they're rare, and the things that happen to the economy in the third quarter of an election year usually look a lot like the things that happened in the first and second quarters.

There are a few other similar points that Trende makes. Some are good caveats for forecasters, and it would generally be good for us to be straightforward about the assumptions of our models. But many of Trende's arguments are simply straw men. Look, we have some models that do a pretty good job explaining elections -- a lot better than claims about campaign quality or candidate optimism or likeability. But past performance does not guarantee future results. Make of these models as you will.


*I hope my post using Silver's numbers to draw out a prediction plot wasn't taken as an endorsement of Silver's model. I simply did it because I thought it would look cool.

2 comments:

Andrew Oh-Willeke said...

The most troubling aspect of the models, which Nate's model compensated for (probably inaccurately), but which there is considerable empirical evidence to support is that ideas and ideology don't matter.

The models are overwhelmingly ideology neutral. It really doesn't matter what the incumbent or challenger proposes to do, even though a large share of voters make an effort to go through the motions of a high school civics model process of expressing considering and weighing policy proposals in deciding upon whom they should vote.

Perhaps the disconnect is that partisan voters do go through this analysis and almost always end up favoring the same party candidate, while swing voters, with whom political junkies are far less familiar, are far less ideological and do act as referendum voters.

Also, perhaps the ideas and ideology piece gets played out as much in what positions politicans take when running for office and while in office. Obama has been no FDR or Kennedy. GWB enacted some of the most expansive expansions of the welfare state and the federal civilian bureacracy of the last couple of decades. Romney enacted Obamacare before Obama did. If the overall issue climate shifts with time due to political campaigning and perceived public opinion, then perhaps merely tracking partisan labels obscures ideological change in which ideas and ideology matter that is still taking place.

Anonymous said...

I do not find the claim about "robustness" to be particularly compelling.

After all, there is precious little variation in the year-to-year vote share of each party. The standard deviation of GOP two-party vote is just 5.9 percent. So why should I get so excited about a model that explains 90 percent of this variation? How much have I actually learned?

I'd extend this analysis a bit further, in two ways.

First, an extremely large amount of variation is a priori excluded by setting the two-party vote as the dependent variable. This is hugely problematic in many respects, and I'll just mention one: it inherently biases the models toward economic explanations, seeing as how the major third-party candidacies of 1948, 1964, 1980, and 1992 all had little or nothing to do with economics (at least not directly).

To quantify the effect of this "two party vote" maneuver, which is rarely if ever justified theoretically, the standard deviation on Republican share of the total vote from 1948-2008 is over 7 percent, but using the two-party vote it drops to a little under 6 percent. This makes for an 18 percent cut in total variation needed to be explained, before we even start the modeling.

Second, I'd note with interest that fully 40 percent of all total variation in GOP two-party vote is in just three elections: 1964, 1972, and 1984. This is a huge problem for any statistical model, especially one with so few variations. It creates enormous inferential problems, which are virtually never discussed, even in the academic literature.

My back of the envelope calculation suggests that more than half of the total variation is accounted for by moving to the two-party vote, then in those three outlying years.

So: why should I care about a model that claims to have an r^2 of 90 percent, all things considered? And especially when the thorny statistical and inferential problems are almost never addressed by the author of the study?

On a personal note, I will said that I have spent the last six years studying the academic literature on campaigns and elections. From 2005-2008 I was obsessed with the political science, quantitative stuff, and for the last three years my focus has been on the historical stuff that is more qualitative. I would say that I have learned so much more from the history than from the political science. If somebody asked me "How do I understand the 1968 election?" I'd point them to Ted White's Making of the President before any quantitative study in the APSR.

Jay Cost