Does unemployment affect midterm elections?

I've seen no small amount of hand-wringing from Democrats worried about the impact of the current double-digit unemployment on next year's midterm elections. By most estimates, even if there is robust economic growth for the next year, unemployment will still be quite high a year from now, maybe upwards of eight percent.* What kind of effect could this have on the election?

Actually, there's not much evidence unemployment has any effect at all. I grabbed the annual figures on unemployment from the Bureau of Labor Statistics and looked at unemployment as a predictor of midterm House seat gains for the president's party since 1950. Here's the scatterplot. The points are labeled by year and the name of the incumbent president.
What do we see here? Basically no relationship at all. Reagan saw very high unemployment in 1982, and his party lost a lot of seats, but no more than the president's party did under Bush in 2006 or Truman in 1950, when unemployment was pretty low.

Okay, so what about the growth in unemployment between the year before the election and the election year? Here's that scatterplot:Again, bubkes. Unemployment actually increased prior to the 2002 elections, but that was the biggest gain for the president's party in this whole time series. Similarly, the biggest loss by the president's party was in 1994, when unemployment had shrunk.

What does seem to matter is economic growth. Here's one measure, the growth in real disposable personal income between the third quarter in the year before the election and the second quarter of the election year. (Data from the Bureau of Economic Analysis.)
It's a noisy measure, to be sure, but it is statistically significant. You can specify income growth several different ways and still get a similar result. Something else that matters is the president's popularity. A ten-point increase in the president's Labor Day Gallup approval rating can save the president's party 16 seats in the House.

So why might income growth matter but unemployment not matter for elections? It's possible that unemployment fluctuations disproportionately affect people at the lower end of the income spectrum -- people who aren't terribly likely to vote in midterms and, if they do, are probably voting Democratic anyway. Growth in disposable income, however, affects everybody, including the moderate voters who will switch party allegiances from time to time.

*Nate Silver estimates the unemployment rate will be around 9.5 percent by next November.