Do Law Schools Report Their Data Honestly?

Just how honest are law schools when they report their data to U.S. News for our 2010 Best Law Schools rankings? Each year, we ask law schools to report the same data to us that they report on the American Bar Association's annual accreditation questionnaire. It turns out the schools are pretty reliable in their data reporting. (Of course, there were some notable exceptions and data errors that I have written about in my blog: Updates to Some Grad School Data and What Happened With Brooklyn Law School.)

The basis for this assessment comes from a study by Tom Bell, a law professor at the Chapman University School of Law in Orange, Calif. He has just published A Model of the 2010 USN&WR Law School Rankings on his blog, Agoraphilia.

Professor Bell says of his work:

"As in every year since 2005, I this year again built a model of the law school rankings published by U.S. News & World Report. Figuring out the rankings—the '2010' rankings, as USN&WR's calls them—proved especially trying this time around. USN&WR changed several parts of its methodology this year and the ABA, which distributes statistical data on which my model depends, fell far behind its usual publication schedule. Finally, though, the model ended up generating scores gratifyingly close to those that USN&WR assigned law schools."

Here's a snapshot comparison of the results of the real U.S. News law school rankings and Bell's simulation of the U.S. News Top 100 law schools.

To what degree did Bell's model duplicate the U.S. News law school rankings? Bell tested its accuracy using "r-squared," a statistical test that measures how close one set of numbers is to another set. It shows that his 2010 edition rankings achieved an r-squared score of .999—in other words, his results were nearly identical to the U.S. News rankings for the scores of those schools in the Best Law Schools' Top 100.

The fact that Professor Bell was able to duplicate our methodology and outcomes using data collected from a different source proves that the U.S. News rankings process is very transparent and accurate.

Professor Bell concludes:

"For now, I'll just offer this happy observation: The close fit between USN&WR's scores and the model's scores suggests that law schools did not try game the rankings by telling USN&WR one thing and the ABA (the source of much of the data used in my model) another. Even a skeptic of law school rankings can find something to like in that."