Wednesday, March 26, 2008

NES screws the pooch

If you use the political knowledge questions from the American National Election Studies, you really ought to read this. Turns out that NES kind of screwed up the open-ended response coding on some of the questions. For example, in 2004, they asked if respondents could identify William Rehnquist. The answer they insisted on was "Chief Justice of the Supreme Court." Rehnquist's actual official title at the time, Chief Justice of the United States, may have been coded as incorrect. So that's bad, and explains why only some 31% of respondents appeared to have accurately identified him even though he'd held that office for almost two decades.

Oh, respondents were also asked if they could identify Tony Blair. Here's the coding instructions for the questioners:
The reference must be specifically to "Great Britain" or "England" -- United Kingdom is *NOT* acceptable (Blair is not the head of Ireland), nor is reference to any other political/geographic unit (e.g. British Isles, Europe, etc.) If unsure whether correct, code as best you can and record R’s response as a remark.
Okay, I'm an Americanist, and even I know that's wrong. Blair, of course, was PM of the United Kingdom, which doesn't include Ireland (although it does include Northern Ireland).

This is pretty damned bad. I use some of these questions when I lecture about how uninformed the average American voter is, so at the very least I've been (slightly) slandering my fellow voters. This is worse, of course, for scholars who have been using these tainted questions in their research to study or control for information effects.

Arguably, it's even worse for our profession in general, since we're supposed to be experts on politics but know approximately dick about the U.S. Constitution and the United Kingdom.


Anonymous said...

This is potentially really bad.

Thank goodness I use the Pew Center's surveys in my classroom when discussing voter knowledges.

Hollander said...

What's really scary is I used the same phrase (screw the pooch) in blogging about the ANES knowledge problem in my What People Know site. Then I do some searching. You used it two days before me.

I feel shamed.

Seth Masket said...

It's a natural coordination point. Don't feel bad.

Did that phrase exist prior to "The Right Stuff"?

Hollander said...

Apparently, according to a couple of web sites, the phrase began with naval aviators. A lot of them became NASA guys in the early days, so I guess it migrated there and then became infamous in The Right Stuff.