Just how honest were law schools when they reported their data to U.S. News for our 2011 Best Law Schools rankings? Each year, we ask law schools to report the same statistical information to us that they report on the American Bar Association's (ABA) annual accreditation questionnaire. Despite some notable exceptions and data errors over the years, it turns out the schools are pretty reliable in their data reporting.
The basis for this conclusion comes from a recent study by Tom Bell, a law professor at the Chapman University School of Law in Orange, Calif. He has just published "Z-Scores in Model of 2011 USN&WR Law School Rankings" on his blog Agoraphilia.
Professor Bell says of his work:
"As I have for each of the past several years, I this year again built a model of the most recent U.S.News & World Report law school rankings. This year's model matched the published rankings very nicely; comparing the model's scores with the published ones generated an r-squared of .997 (where 1.0 would indicate perfect correspondence). Why do my fellow rankings geeks care about z-scores? In brief, these z-scores measure how well each school performed relative to its peers, thereby establishing its rank. Because U.S. News uses z-scores to rank law schools, so too must any model of its rankings. I weighted these z-scores simply by multiplying the z-score for each school, in each category of data, by the percentage that category influences a school's overall score in U.S. News rankings."
To see the weights and factors Bell used in his ranking calculations, check out the U.S. News Best Law Schools ranking methodology. Bell's blog also includes links to z-scores from previous years.
To what degree did Bell's version of the U.S. News law school rankings model duplicate the actual rankings? Bell tested his rankings model's accuracy using "r-squared," a statistical test that measures how close one set of numbers is to another set. Bell's 2011 edition of the U.S. News law school rankings achieved a nearly perfect r-squared score. In other words, his law school rankings results were nearly identical to the U.S. News rankings for the scores of those schools in the Best Law Schools' top tier.
The fact that Professor Bell was able to duplicate our methodology by using law school data he obtained directly from the ABA proves that the U.S. News rankings process is very transparent and can be duplicated using publicly available data. This exercise also establishes that U.S. News is calculating and weighting the ranking variables as stated in the posted methodology. Users of our law school rankings can be confident that the results are correct given the weights and rankings variables that U.S. News has chosen.
Finally, does Bell's study also prove that law schools are being accurate in how they report their statistical profile data to the ABA, the general public, and U.S. News? No, this only proves that law schools are being very careful to report the same data to U.S. News that they report to the ABA for accrediting purposes.
Searching for a grad school? Get our complete rankings of Best Graduate Schools.
Reader Comments Read all comments (4)
Fed up of NH 2:29PM February 09, 2011
K of MD 8:51PM January 20, 2011
J-Dog of IN 12:54PM December 30, 2010