When Republicans woke up to defeat on Election Day 2012, it was clear that they had been outfoxed and outmaneuvered by President Obama and his campaign. Now it's clear why.
Under the political radar, Team Obama made an unprecedented effort to understand the voters and speak their language, slicing and dicing the electorate with a sophistication and savvy that the Republicans couldn't match and are still scrambling to replicate.
The result was the rebuilding and in some cases expansion of Obama's original coalition from 2008, and a remarkable comprehension of what really motivated voters in 2012. I describe the secret Obama plan for achieving that goal, with details never before disclosed, in my new book, "Prisoners of the White House: The Isolation of America's Presidents and the Crisis of Leadership."
Team Obama devoted unprecedented attention to what the president's strategists called "the ethnography project," studying the electorate in exhaustive and minute detail. In the end, Obama and his strategists had a lot better understanding of the public, especially Middle America, than the strategists of Republican nominee Mitt Romney.
They also managed to generate a strong turnout among key constituencies that remain the cornerstone of Obama's political strength to this day, including unmarried women, Latinos, African-Americans and young people. Republicans drastically under-estimated Obama's potential for appealing to this new coalition, and they were left behind relying on old-fashioned turnout models for white voters that, it turned out, couldn't put Romney over the top.
The Obama team's opinion research was led by Joel Benenson, a tough-minded pollster from New York. The bearded, cerebral Benenson is a former beer distributor and ex-newspaper reporter for the New York Daily News who prides himself on his ability to break out of the mindset and insularity of Washington, D.C.
In 2012, he succeeded, largely because the depth of his research was so extraordinary. Benenson says his goal as a pollster is "to understand the hidden architecture of opinion" and to "probe deeply into the underlying values and attitudes that shape how people are viewing the issues of the day and the content of their lives."
One way that Benenson set the Obama campaign apart was through the ethnography project. It was designed as a deep dive into the world of everyday Americans not only to clarify their views on politics but to find insights into their "daily lives," Benenson told me.
In the spring of 2011, Benenson selected 100 "up for grabs" or undecided voters in the battleground states of Colorado, Florida and Ohio. These voters were asked to go to a special website and answer 8 to 10 questions about their everyday activities on six occasions over a 16-day period.
Each respondent took an average of an hour to fill out the questionnaire each time, an unusual commitment of time and energy. The topics ranged from explaining the type of community the participants lived in; how they were treated at work, and how they were adjusting their behavior to cope with the economic downturn.
After the responses were analyzed, nine voters were chosen from among the participants in each of the three states, and they were further divided into groups of three, or "triads." At that point, detailed interviews were conducted to learn even more about them as individuals.
They were questioned, for example, about their routines, their families, their concerns about the present and their hopes and fears about the future. Each of these sessions lasted about 2 1/2 hours. They were also asked whether Obama deserved to be re-elected, and why.
Benenson says this information, compiled into what he calls "ethno-journals," was combined with the results of many regular opinion polls and focus groups. The ethnography project produced 1,400 pages of transcripts and data.
Among the findings, Americans felt that the economic crisis didn't happen overnight and would take a long while to end, probably at least five years. Also, Americans thought that the economy was not undergoing an ordinary recession but a particularly harsh and deep one that indicated the existence of fundamental long-range problems. Obama incorporated these concepts into his campaign message, and he seemed much more connected to everyday Americans than Romney.
Summaries of the research were given to Obama and his key advisers at the White House and within his re-election campaign. It all amounted to a rich resource that the Obama campaign and White House officials used through the rest of Obama's first term, in the fall campaign, and today. "You've got to hear the conversations that people are having" or you can't solve their problems as a political leader, Benenson told me.
All this research was one reason why Obama injected certain themes into his presidency and his re-election campaign, such as the idea that he was attempting to create an economy that was "built to last" and that should be constructed from the middle up, not from the top down.
It was also important for Obama to acknowledge the harsh facts of economic life that most middle-class Americans were experiencing. For them, the economy had improved somewhat but was still causing many hardships and strains. So Obama talked about how the economy had "turned a corner," and he argued that his policies were starting to work, but stopped short of saying that his policies had fully succeeded.
This is the reality that Benenson found in the lives of Middle Americans, and they came to believe that Obama understood them far better than Romney did.
Fast forward to today, and it's clear that the Republicans are still trying to find ways to pull themselves out of the trough. Reince Priebus, the chairman of the Republican National Committee, recently issued a report that gave a scathing assessment of the GOP's troubles and acknowledged how far behind the party had fallen in understanding and appealing to the electorate.
The party has lost the popular vote in five of the last six presidential elections (including 2000 when George W. Bush lost by 500,000 ballots nationally to Democrat Al Gore but won the presidency after the Supreme Court awarded him the disputed state of Florida).
And the future doesn't look bright in view of the GOP's unpopularity with several growing constituencies, including unmarried women, Latinos, African-Americans, and young people.
But the story of Obama's ethnography project is also a cautionary tale for the president. It's not enough to understand the electorate. A president also must be able to act on that understanding and make progress, and that's where Obama has fallen down on the job. His agenda has stalled, and in the case of a gun-control bill, it has been defeated outright on Capitol Hill.
At his news conference this week, Obama admitted that, 100 days into his second term, he was having trouble getting things done. He blamed Republican intransigence and said he can't be expected to make his opponents "behave."
"Right now, things are pretty dysfunctional up on Capitol Hill," Obama declared, stating the obvious. But Americans expect more than excuses. They want their president to make the system work. They want their leader to be not only a good listener, but also someone who delivers results.
Ken Walsh covers the White House and politics for U.S. News. He writes the daily blog "Ken Walsh's Washington" for usnews.com, and is the author of "The Presidency" column for the U.S. News Weekly. His is the author of the new book "Prisoners of the White House: The Isolation of America's Presidents and the Crisis of Leadership," on which this report is based. He can be reached at email@example.com, and followed on Facebook and Twitter.