Desperately seeking trends

Doug Schoen and Pat Caddell's latest piece of advice to Obama suggests that his campaign is in trouble and needs a sudden change. As evidence of Obama's recent troubles, they cherry-pick from a number of polls, noting that Obama is trailing Romney badly in the "battleground" state of Missouri (which isn't actually a battleground this year) and that Romney has closed the gap with Obama nationwide since May. Then they do this:

President Obama now leads by just one point in the latest PPP Florida poll (48%-47%)—down from a four-point lead (50%-46%) in an Aug. 22-26 CNN poll.

Look, I'm no pollster, but isn't comparing results across different polling firms and calling it a trend one of the cardinal sins of polling interpretation? And, given those polls' margins of error, aren't those two results statistically indistinguishable from one another?

Not "scientifically valid"

Sounds like some of the folks at the Douglas County (Colorado) School District could use a statistics refresher course. Writes the Denver Post:

The Douglas County School District has deemed its spring survey of parents "inconclusive" — a poll in which a majority of responding parents saw the district's suspended voucher program as "unfavorable" and expressed unhappiness with the district's overall direction.

District officials contend that not enough parents participated in the survey to make it a valid representation. As a result, they stamped every page of survey results posted on the district's website in red letters saying: "Inconclusive due to insufficient response rate."

More than 4,900 parents completed the survey. There are approximately 76,500 parents whose children attend Douglas County schools. 

[...]

District spokesman Randy Barber said the approximately 6 percent of parents who responded to the survey this spring was significantly under the 30 percent the district wanted to make it "scientifically valid."

Yes, people usually have more confidence in a survey if it has a higher response rate. But that doesn't mean that the results of this survey aren't valid, and there's nothing magic or "scientific" about the 30 percent threshold. We can conduct very accurate national surveys based on just 1,000 responses -- roughly a fifth of what they managed in this parent survey. More important than overall numbers is representativeness. That is, does the sample of parents who responded to the survey look roughly like the overall population within the school district? This can probably be figured out using demographic questions (although the only one I can find in the survey is just a race question. Area of residence, income, family size, ideology, etc., would be really helpful here.) 


Of course, maybe the reason that the district rejected the results as "inconclusive" has less to do with sample size than it does with the results, which were less than flattering for the district. If they really want to know what district parents think, they could conduct a survey of a representative (and probably smaller) sample and get solid results. But if they just want certain results, then by all means they should just keep doing surveys until they get the answers they want.

The big "O" stands for "omitted variable"

Yeah, about that Match.com survey (via John Sides)...

Republican Lead the Polls—In Orgasm

Yep, you heard that right. Republicans—and conservative Republicans, for that matter—reported the highest frequency of orgasm of all of the survey respondents, despite having the least amount of sex. More than half of those who identified as conservative Republicans said they reached climax almost every time they had sex, compared with just 40 percent of liberal Democrats. Sure, these answers are self-reported, but the survey was conducted anonymously online. What reason do they have to lie?

Well, there are plenty of reasons people would lie in an anonymous survey, particularly about such a subject. But did they think to control for gender?

Ideology                        Percent male

Extremely liberal                  37.3
Extremely conservative        53.2

(Source: ANES 2008)

What a shock -- the percentages who claim to have orgasms every time they have sex are almost the exact same percentages who happen to be men.

Romney in Two Primaries

Here's a graphical comparison of Romney's performance in the Florida primaries of 2008 and 2012 among different demographic groups:

The red line is where Romney's support would be if he'd done the same in 2012 as he did in 2008. Obviously, he did much better overall, garnering an estimated 46% (as of 11:30 EST) as compared to 31% four years ago. And he improved in all the subgroups I've charted above. But he improved in some quite a bit more than in others.

He seemed to make the greatest inroads among the wealthiest voters, Catholics, women, the elderly, and those who believe the economy to be the most important issue. He made his weakest gains among Evangelicals, pro-lifers, men, and Protestants.

The gender gap strikes me as an interesting story this year. There was an enormous gender disparity in turnout in 2008, with men outpolling women 56-44. This year, the electorate was just about even between men and women. In 2008, meanwhile, Romney got 32% of men's votes and 30% of women's -- not much of a difference. This year, Romney got 41% of men and 52% of women. (Gingrich seems to have attracted men to his campaign much more than women.)

Well, the math is easy

I suppose they're obligated to show us a race cross-tab in the South Carolina Republican exit polls, but still...

F-ing polls -- how do they work?*

Roger Simon has written one of the most anti-intellectual columns of the week, asking whether polls are really "magic." Not only does he appear not to know how polls arrive at the answers they do, but he seems to have no interest in learning. He even falls back on the classic "they never call me" trope. Some highlights:

I have never been called by a political pollster and don’t know anybody who has, but I know some pollsters, who assure me they don’t make the numbers up, and I believe them.

Pollsters, or rather the phone-bankers who make call after call (or computers that make robo-call after robo-call) do get people to talk to them. Not vast numbers of people, but pollsters do not require vast numbers.

[...]

We are a nation of nearly 313 million people. So how many people did the pollsters actually speak to? If you have extremely good eyes, you can find the answer in tiny type at the bottom of a chart: The Post-ABC poll was conducted by phone “among a random sample of 1,005 adults.”

That represents 0.0003 percent of the nation at large. (The number of Republicans and Republican-leaning independents was an even smaller sample of 395 people.)

[...] 

This poll has a very good reputation and I “believe” the results in that I believe they were calculated carefully and (unlike some partisan or campaign polls) without any agenda.

[...] 

Does Obama really lead Gingrich by 8 percentage points in a (currently) imaginary matchup?

I dunno. Sounds right to me. But I am an even smaller sample than 0.0003 percent.

You really don't need to be a statistician to understand this stuff. Why can a survey of 1,100 people be accurate in telling us how the whole nation is thinking? The metaphor I always liked was a blood test. For a doctor to determine if there's a problem with your blood, she doesn't need to remove it all -- she can just extract a small vial. This vial of blood represents the rest of your blood well because it's constantly being mixed up, so that a few cc's of your blood in your arm looks like the blood anywhere else in your body.

It's the same thing in surveys. You can poll a fairly small number of people as long as you can be confident that you're getting a representative sample of American voters. (Talking to your friends and neighbors? Not representative. Calling people randomly across the country? Much better.) And some relatively simple math can tell you just how likely it is that your sample believes what the rest of the country believes. Picking 1,100 people for a survey means you have a margin of error of roughly 3%. That means there's a 95% chance that the actual population is within three percentage points of what your sample believes. Pollsters have settled on that as a pretty reliable margin. You could get it down to 2%, but only by interviewing lots more people, driving up the costs of the poll considerably without improving its accuracy by much.

The sad thing is that Simon has an audience who might really appreciate a better understanding of how polling works, but he decided to waste their time with some blather about how polls are magical and therefore beyond our understanding. They're not, and Simon's readers deserve better.


*Must credit Brendan Nyhan for the Insane Clown Posse reference.

Do good politicians need help?

This post by Gladstone at Cracked is a bit old, but contains some important notions about presidential popularity that deserve to be addressed. Gladstone is discussing the website "WTF Has Obama Done So Far," a tongue-in-cheek project designed to demonstrate that Obama's presidency has, in fact, been marked by a great deal of accomplishments of which liberals should be proud. One would hardly expect such a website to have an enormous impact on elections or public discourse, but Gladstone actually takes it as evidence of Obama's failure as a politician:

A true presidential politician articulates a vision for America, wins over the public support and then gets the Congress to follow him because everyone wants to be on the winning team. And by that standard, Barack Obama has not met his objectives. How do you know? Because after two years in office, he needs the liberal devout to engage in an Internet campaign to explain what he's even done. [...]

[T]wo years into Reagan's term no one had to be told what he did. It didn't even matter if it were true. Americans would tell you Reagan cut inflation, made us stronger abroad and restored our national pride. Furious, the liberal intellectuals would then take to the media to explain why Reagan's seeming accomplishments were a smoke screen. Why he had taken credit where none was due. And point out all the unforgivable things he hadn't done or did in secret. Meanwhile, the right would merely smirk at those brainiac, detail-orientated liberals, while mumbling things like, "There you go, again," because they knew they'd already won.

Gladstone is funny and compelling, but also profoundly wrong about a few key items here. First of all, the obvious one: Reagan was deeply unpopular two years into his first term! In January of 1983, Reagan had approval ratings in the mid-30s, well below Obama's lowest point thus far. That doesn't make him a bad politician or a poor communicator; it's just a reminder that presidents are, to a great extent, victims of circumstance. The economy was in the middle of a full-blown recession, and that takes a toll on even charismatic politicians like Reagan.

Second, how did Reagan manage to win reelection in 1984? It's simple: the economy recovered. If "no one had to be told what [Reagan] did," that's because everyone could experience economic recovery in their own lives. More people were finding work, employed people were making more money than they were the previous year, their buying power wasn't being wiped out by high inflation, etc.

Third, even if people didn't have to be told what Reagan did, they were told anyway. No, there were no sarcastic websites in 1984, but there were plenty of conservative operatives at work getting that message out. They could largely be found in places like the White House, the Republican National Committee, the Reagan/Bush reelection campaign, and more than a few newspaper editorial desks, and they spent a great deal of time, money, and energy telling Americans that their lives were better because Reagan was in charge. In other words, they were doing exactly what liberal activists are trying to do for Obama today.

If most Democrats can't recite a laundry list of Obama's accomplishments, well that's just because very few voters can do that about any president, not because Obama's a bad politician. And if economic growth actually ramps up in the next year, Gladstone will be amazed at how good a politician Obama suddenly appears to be.

The Plunge

Firstpost:

Obama poll numbers plunge ahead of job speech 

Jonathan Bernstein:

It just isn't happening. What has characterized Barack Obama's approval ratings since about September 2009 has been stability with very gradual deterioration over time. Certainly not a "plunge" at any point beyond summer 2009. That summer, if I recall correctly up until mid-August, yes. Since? Nope.

Bad spin can't save a bad poll

Last week in Wisconsin, Democratic state senator Jim Holperin survived a recall attempt, winning the race by a healthy 55-45 margin. Automated pollster We Ask America didn't do such a great job forecasting the race, though, showing Holperin only holding on 51-49. Now, to be fair, it's really hard to forecast these off-year, off-season recall elections -- you never know who's going to show up (although PPP was pretty close). But We Ask America's COO Gregg Durham decided to defend his organization anyway:

It's so hard to tell when you do one poll. One thing you can't judge is what the turnout will be. In this case, unions were heavily involved in turning out Democratic votes. Now, I will stand by the numbers -- this may be what the general electorate wanted, but not what the people who turned out wanted.

A pollster should know that those who turn out to vote are the general electorate. I'm not sure what group We Ask America surveyed, but it wasn't the general electorate.

The predictive power of polls

If you're already looking at polls for the 2012 presidential election, be sure to glance at the awesome graph below from Chris Wlezien and Bob Erikson, courtesy of John Sides. The chart depicts the R-squared of polls predicting outcomes in presidential elections. R-squareds range from 0 to 1, with 1, in this case, meaning that polls perfectly explain what will happen in the election and and 0 meaning there is no relationship between polls and the election outcome.

Notably, we are still currently more than 300 days away from the next presidential election, meaning that any polls pairing up Obama against possible Republican nominees are functionally meaningless. But enjoy them anyway.

A nation of Betazoids

Recent Fox News poll question:

Do you think President Obama is truly serious about reducing the budget deficit, or not?

So now we're asking the American people to assess the intentions of public officials? Is this really something they're equipped to do?

For what it's worth, 51% of respondents think Obama is "truly serious," while only 38% and 42% think Congressional Democrats and Republicans, respectively, are "truly serious." A majority also felt the Romulans had a hidden agenda.

Even you cannot avoid... pressure

The Tarrance Group has put together a weird little poll (PDF) on health care reform conducted in a bunch of swing congressional districts across the country.  The big question is this one:

Do you favor or oppose the health care reform legislation being proposed by President Obama and the Democrats in Congress?

In all but one of the districts, a majority oppose the legislation.  58 percent claim to oppose it in Betsy Markey's district (Colorado's 4th CD).  And roughly 3 out of 4 voters say their member's vote on this bill will be important to them in November.  So you can get a sense of the pressure being put on folks like Markey right now, who want the bill to pass but don't want to be the one to pass it.

That said, I have no idea how realistic this sort of question is.  Will voters really punish swing Democrats for voting for health reform?  Would they really reward those members for voting against it?  How much will this matter compared to, say, the status of the economy or any other big issue that might arise between now and November?

Even less realistic are some other questions in the poll, such as:

Would you favor or oppose the Democrats in Congress going outside of normal procedures and using a little used parliamentary tactic known as “reconciliation,” to pass a health care bill, over Republican objections? 

Question wording bias, anyone?  Strong majorities say they oppose reconciliation in this poll.  Okay, forget, for a moment, the lies built into this question (reconciliation is a normal procedure, it's not a "little used tactic," it wouldn't be used to pass health reform anyway, etc.).  Who the hell will care about congressional procedures six months from now?  It's not remotely credible.

Oh, by the way, roughly 3 in 4 respondents believe that their taxes will go up as a result of health reform.  Another lie, but that's what the swing Democrats have to deal with right now.

(Thanks to Kyle S. for the link.)