The problem with this question is that the answer, like the role of racism in American politics in general, is deeply nuanced and doesn't lend itself easily to an A or B response. Ari and I had a spirited discussion on this very topic last week (see the comments here).
Jimmy Carter is making a causal argument: Obama's race is causing many conservatives to oppose him viciously. Of course, any causal argument implies a counter-factual one: if Obama were white, many conservatives would not be opposing him viciously. This, to my mind, strains credulity. Anyone who thinks a white Obama would not be encountering this kind of resistance just doesn't remember Bill Clinton's presidency. Jerry Fallwell called Clinton a murderer. Jesse Helms suggested that Clinton would be shot should he visit the Carolinas. A parody Clinton/Gore bumpersticker read "Commie/Bore." Hillary Clinton was, of course, called a murderous lesbian, as was Janet Reno. And, in the end, Bill Clinton was impeached. And by the way, he was white.
Now this doesn't mean that race has nothing to do with what's going on today. We obviously see a lot of racist crap showing up at protests and town hall meetings. And when white people scream that they want their country back, it's hard not to think that race is playing a role there.
One thing that was both surprising and gratifying was the relative lack of a race effect in last year's election. John Sides did a nice analysis showing that Obama's race may have hurt him by about one percentage point. That's not nothing, but it's far less consequential than many had expected earlier in the year. Of course, that's just another counter-factual argument -- a white Obama would have done about the same as the black Obama did. But this doesn't let white racists off the hook. The two major parties have polarized on racial issues in recent decades, to the point where most white racists, if they vote, are voting Republican. (No, I am not arguing that Republicans are racist. Just that most racists today are Republicans.) So of course Obama didn't lose votes because of his race. He was never going to get those votes anyway because he's a Democrat.
I am confident that either Hillary Clinton or John Edwards would be facing similarly virulent opposition today were either of them president. Probably the same folks bringing the racist drivel to protests would be spewing misogynistic or gay-baiting slogans today. What's motivating the anger is simple ideological resistance to big government (properly distilled by conservative opinion-making elites). It has the added feature of bringing out the racists, but I don't think that's most of what's going on here. As I suggested earlier, you can explain much of what's going on without reference to race.