Who's Afraid of Larry Summers?

Here's the people who aren't exactly excited Summers is the frontrunner to be Fed chairman.

By + More

Whoa, what just happened? In the last day, Larry Summers went from another face in the pack in the race to replace Ben Bernanke as Federal Reserve chairmanrace to the front-runner.

The buzz has hit fever pitch, and the White House has even weighed in, in its own indirect way. President Obama reportedly "really likes" Summers, in the words of the Washington Post's Ezra Klein, whose Tuesday-afternoon blog post amped up the Summers buzz. CNBC's John Harwood also heard from a source close to Obama that Summers "should be" the Bernanke successor.

[READ: The End of the Easy Money Romance]

Still, it doesn't matter if the president really likes his pick if other people don't get behind him. Below, U.S. News has broken down the people who might oppose a Larry Summers-led central bank, and why.

1. Yellen boosters

Until the last few days, Janet Yellen was the leading name in the handicapping for a new Fed chairman. There are plenty of reasons why she had been favored: For one, she's almost there already, as the Vice Chairman of the Federal Reserve. She's also very qualified, having headed the San Francisco Fed and Council of Economic Advisers before she became Ben Bernanke's No. 2.

Yellen also has plenty of smart people on her side. Former FDIC chair Sheila Bair penned a strong endorsement of Yellen in Fortune this week, saying that "there is no better qualified candidate" than Yellen and noting that she was one of the earliest central bankers to recognize the risks of the sub-prime mortgage crisis.

Summers, as a former Treasury Secretary under Clinton and chairman of the National Economic Council under Obama, has his own impressive resume, of course.

"He's very knowledgeable, he understands the system, he understands how financial markets work, he has a real network in both Wall Street and national politics," says Robert Shapiro, chairman of economic and political advisory firm Sonecon and a former under secretary of Commerce for economic affairs. However, Shapiro adds, Yellen is also very knowledgeable, and she comes with the bonus of nearly a decade of Federal Reserve experience.

"[Summers'] experience is not in that context, as opposed to Janet Yellen," he says.

2. Deregulation opponents

Larry Summers was part of the Clinton-era economic team that introduced the Gramm-Leach-Billey act, as Quartz pointed out last week. That law, which removed barriers between investment and commercial banks, was one contributing factor to the financial crisis.

It would be "patently absurd" to pin the crisis on Summers, says Shapiro, as plenty of high-ranking government officials in both Congress and the executive branch pushed for deregulation.

[FLASHBACK: Obama Considered Summers to Lead World Bank]

Still, Summers' position on regulation has set off alarm bells for some. The former Treasury secretary has not moderated his views on financial regulation since the crisis: He has on multiple occasions defended his role in deregulating banks during the Clinton era. For that reason, Summers is not a top prospect to lead the Fed among those who want a closer watch on banks.

"The Fed, now more than ever, is the linchpin going forward of financial regulation, and so I want someone who has laid down some real markers that 'we take a skeptical eye toward activities in finance,'" says Josh Bivens, research and policy director at the Economic Policy Institute, a left-leaning think tank.

3. Women

OK, this is overstating the case. By no means are all women watching eagerly to see who becomes the next Fed chairman, nor are all women going to remember Summers' now-infamous 2005 remarks as president of Harvard University, when he gave a speech implying that women and men are biologically wired to have different scientific and mathematical aptitudes.

But for those who do remember, the sting remains. Women's advocacy group UltraViolet has released a statement arguing against a Summers pick:

"Summers has a long history of making sexist comments and the Federal Reserve has so much control over our economy – an economy that's hitting women and families especially hard right now"