Grad Rankings: Share Your Feelings

Reader Comments
Back to blog

Mr. Morse
I have two related concerns with the current ranking methodology for law schools:
1. Employment data -- Including employment data would seem to influences the outcome of future employment prospects. I'd be interested in seeing a statistical study about how the rankings influence hiring and recruitment decisions. Most large firms limit their recruitment to higher-ranked schools, and a hiring partner's eyes would likely gravitate towards a top-10 school's graduate compared to a 3rd tier school's graduate. If employment data is included in weighting, then it could influence the employment data for the following year's rankings.
2. Hiring partner and judges surveys - You stated that you mainly asking hiring partners and judges-those that hire new minted J.D.s. Similar to employment data, these surveys become perpetually biased if there is no standardization of what the hiring partners and judges know about each school. Have they met the professors or school administration? Do they know the current bar pass rates of the schools? Do they all know or currently work with students at the schools they rank? Are the surveys only done by those local to the schools? Because someone knows the name of a school and it has "a good reputation" does not necessarily mean that a schol with less name recognition isn't worthy of "a good reputation."
The overarching issue for including both of these measures of quality is that they strongly influence *future* data used to calculate rankings. This is a case of the statistical study having influence upon those being observed (i.e. the survey takers). It's akin to saying to the survey-takers, "Here are the best schools from last year. After seeing that, rank the best schools for this year." A more scientific and unbiased methodology would attempt to have minimal influence.
A separate concern that I have is that your weighting only 0.02 for bar exam passage rate. The relative rankings for schools are absurd because of the low weighting for this measure of quality. The rankings purportedly measure the quality of students entering and graduating the school with LSAT, GPA, bar pass, and employment data. The quality of the students within and graduating from the school should give the greatest measure of the schools' quality. The bar exam is purported to assure the competency of lawyers. Thus having a higher pass rate presumably assures higher overall competency of students becoming lawyers.
I graduated from the University of San Francisco. USF is *Third Tier* this year with 85% pass rate for CA July 2007 exam. I know July 2007 exams may not reflect in the rankings until next year, but USF's third tier ranking is extremely different from following top-100 schools (including one nearly 100 rankings higher with a lower bar passage rate)?
It had a bar pass rate relative to:
Boalt Hall (ranked #6 with 82%)
University of California –Hastings College of The Law (#38 with 81%)
University of California - Davis (#44 with 84%)
Pepperdine University School of Law (#59 with 78%)
Loyola Law School-Los Angeles (#63 with 78%)
Santa Clara University School of Law (#77 with 76%)
University of San Diego School of Law (#82 with 80%)
McGeorge School of Law (ranked #95 with 73%)
I look forward to your response. Thank you!
Bob Morse responds: Your post is long and thoughtful. Difficult to respond to in brief. You are correct that we should consider boosting the weight of the bar passage rate since it's a good outcomes measure. U.S. News has not done any statisitical studies about the impact of our rankings on hiring and recruiting, I know that some law professors and others have. Maybe they will comment on my blog. We do need to study ways to improve the lawyer/judge survey.

Michael of CA 7:24PM April 10, 2008

Mr. Morse, I think Brian Leiter's open letter suggests adjustments to your ranking system which it would be unreasonable for you not to accept. I hope that you respond to his letter publicly without further delay.
Bob Morse responds: Brian Leiter's letter ( http://leiterlawschool.typepad.com/leiter/rankings/index.html) does deserve a thoughtful response. The response will be made at the proper time and in a proper forum. However, it should be noted that he recommends almost a complete change in our ranking methodolgy, how we conduct our data/survey collection and the data we use in the rankings.

JIK of MD 7:16PM April 10, 2008

Mr. Morse,
thank you for taking the time to respond to my post. However, what i had suggested was not revealing the "individual" responses but simply to whom the survey was mailed and who of those people returned the survey.
It does not have to be a "mailing list" per se but maybe a graphical representation by state on who was invited and who participated. i.e 23 surveys mailed (XX federal judges, XX managing partners in large law firms and XX law school deans in connecticut, 11 surveys returned (XX state judges, XX managing partners, XX state officials) from connecticut. metro areas like NYC or Boston could be demarcated apart from the states.
So if the Law school in sparsely populated midwest state gets a low rating, people who want to practice in such state could draw a proper inference as to the rankings.
Also I wanted to know your thoughts on adding a weighted category that accounts for how long a law school has been accredited (because one could reasonably infer that the longer a school has been accredited, it's "influence" on the legal community would be greater).
Bob Morse responds: We said that we survey 4 academics at each law school (a total of 780 surveys) and we say what positions we survey-Dean, dean academic affairs; chair fac appt. committee and most recently tenured faculty member. We publish our 70% response rate in peer survey. We surveyed 643 hiring partners and judges and got a 26% return rate. If we publish the details that you ask, we risk revealing which surveys have been returned. We have said that we don't know who does the actual rating, we only know that a survey that was mailed to particular person has been returned. Also, we think the accreditation issue is covered via the peer surveys.

neil of CT 9:06PM April 08, 2008

The last set of rankings for graduate schools in the areas of Social Sciences and Humanities was 2005. When will those rankings next be updated?
Bob Morse responds: There will be new rankings in those PhD fields in the Social Sciences and Humanities published in March 2009.

Josiah Daniel of 2:46PM April 08, 2008

I don't think there can be any argument that the rankings are and will continue to be a very important part of the law school application process. Throughout my application season, they have been a useful guide to me and incredibly helpful in my considerations. Rather than debating whether the rankings should exist or not, we should help USNews to find ways to improve them.
As a rising 1L at a top five school, I worry about how broad the rankings are. Perhaps the best way to improve the rankings would be to pare down the number of categories under consideration, emphasizing those categories which are most universal, and least open to manipulation. For example, bar passage rates from one school to another should be expected to vary from state to state to such an extent that I think they are not useful. The employment statistics also seem to be so open to finessing that I would argue they should be dropped. Expenditure numbers are another area. I wonder also, in this digital age, how relevant it is to count the number of titles in a library when most students will use westlaw for their research?
The USNews rankings are relevant and will continue to be so, especially as long as Mr. Morse and company are seen to be making a good-faith effort to keep them free of manipulation and as relevant as possible for prospective students.
Bob Morse responds. We do adjust resources for regional cost-of-living differences. Disagree very strongly that we should drop the career (job placement data) and bar passage rate data.

Greg of MD 11:35AM April 08, 2008

Dear Bob
I am a practicing lawyer whose son is going to law school next year.
I am concerned because law schools are "gaming" your system and it creates some problems. For example, law schools waive admission fees and recruit students to apply they have no interest in to boost their selectivity rating. You should at least try and find what % of applicants the waiver of fees applies to at each law school and adjust selectivity accordingly. (No credit should be garnered when fees are waived because real selectivity is gleaned by those who wanted to apply and paid for it .)
A second concern is facuty-to-student ratio. I think it is more important the number of full-time faculty-to-student ratio. Part-time (adjunct) professors could teach three students and skew the result.
I think reputation is scary if you do not know that the people surveyed have actual knowledge of who they are rating. Let's be clear, law schools rank students they have no idea about so I do not object to rankings, but they should be as reasoned as possible. You should try to confirm that those filling out surveys on law schools have actual knowledge of the schools they are actually commenting on. A smaller survey leads to skewed results so if there is a way to increase the pool when it is too small you should do so. (In polling data, a small survey has a lot more room for error.)
Thanks for thinking about this.
Bob Morse responds. Thanks for your comments. The student faculty ratio usnews publishes is what the law schools report to the ABA. In terms of the lawyer judge survey, we mainly asking hiring partners and judges-those that hire new minted J.D.s.

rsbrown365@comcast.net of MD 8:00PM April 07, 2008

Bob,
What about instead of "job satisfaction" as the commenter above suggests, USNWR started to track placement in legal-related positions? Some of this data is readily available from just looking at the career services web sites of many schools (e.g., - % in law firms). While it's true that some law students take the JD and use it toward non-law related goals, this would still be a meaningful statistic for students who seek a school that prepares legal practitioners rather than government policy wonks.
Another useful statistic, though I'm unsure how to track it, would be temporary versus full-time, permanent employment. Technically, contract attorneys are in a law-related practice, but I imagine very few of them would consider document review under deplorable conditions the job which they pursued a legal education to perform.
Finally, I would propose including a statistic which you already take into account with your undergraduate rankings: alumni giving. Law schools may take second place to an alum's undergraduate institution in donations, but alumni giving to a law school is a reflection of the positive experience a student had there and their financial stability post-graduation. This also can show the sense of community present in a law school.
Bob Morse responds. Thanks for your good ideas. In my view job satisfaction is not available and usnews would get access to graduates to measure it. The other areas, especially alum giving, we could consider.

James of MA 8:57AM April 07, 2008

Sir,
I am currently a medical school student and I strongly disagree with the methodology used to rank medical schools. If I read correctly it says on this website that, "All the health rankings are based solely on the results of peer assessment surveys sent to deans, other administrators, and/or faculty at accredited degree programs or schools in each discipline."
I cannot understand why the rankings of all medical schools are based purely on peer assessment. Granted, these peer assessments are from respected deans and administrators of schools but I think that it is impossible to understand the full capabilities of a school based solely on these people's opinions. Your methodology fails to include two factors that are testaments to a school's ability to produce great doctors. These two factors include the student's average USMLE Step 1 and Step 2 scores at each school and the number of students accepted into competitive residency programs. Because of the different grading systems and different methodologies of teaching at each medical school, the only objective means to measure a students' knowledge of medicine is through the Step 1 and 2 scores. Thus all residency programs use the Step 1 score as an objective way to assess medical students' knowledge after the first two years of medical school. Therefore, why can't the US News not use it as an objective measure to rank medical schools? The Step 1 and 2 scores are a great way to measure how well the students at a school are performing academically. In addition, residency programs in dermatology, neurosurgery, plastic surgery, and orthopedic surgery are extremely difficult residencies to get in to. Therefore, the number of students accepted into these programs each year should be a sound testament to how well the school is doing in producing competitive and competent medical students. Yet, there is never a mention of this in your ranking systems.
The US News and World Report's graduate school ranking is extremely influential. As a pre-med student I had used it to guide my decision when it came to choosing a medical school and so did many of my other friends. Therefore, it saddens me to see that its methodology is extremely flawed and it fails to truly grasp how well a medical school's academic program truly is.
Bob Morse responds. You are very incorrect. The medical school rankings are not based solely on peer surveys. Go to this link and read the medical school ranking methodology: http://www.usnews.com/articles/education/best-graduate-schools/2008/03/26/medicine-methodology.html

Thanh Tran of TX 3:30AM April 07, 2008

Is there a second tier in the law school rankings? I only see tiers 1, 3, and 4??? If so, at what rank does it begin, 50?
Bob Morse responds: There is not a tier 2. It's just the Top 100. In fact the phrase Tier 1 will soon be removed from the web site.

bill of CA 12:28AM April 07, 2008

Mr. Morse,
as an experienced Army Officer (over 10 ten years of service to include recruiting,retention, training, systems fielding, contract awarding and mangement) i believe i understand statistics and ranking as well as any expert. In my experience, most bureaucrats find ways to "game" the system especially where some facets of the process are anonymous.
as i read the artice in the ABA Journal, i wanted to know why US news continues to keep the names of people who participated in the "prestige" portion of the survey as secret. I think publishing the names without revealing what the person said on the survey would allow proper "self reflection" by law school deans who complain about your methodology. furthermore, it would also allow prespective students to realize that a local / regional school is ranked low because all the people who returned the "prestige" survey were concentrated in a specific regional area or were from a limited number of "top ranked" schoools.
on a personal note, when i picked a law school to study in the ranking did not make a difference, i used the LSAT/GPA filer to pick realistic choices for applications. I picked the law school that i attend now over other schools because the Dean of Admissions personally called me to inform me of my selection.
v/r
Neil
Bob Morse responds: We have promised the respondents that we will not release individual responses and that we will publish only summary scores, and we plan to keep our word as journalists. It is reasonable to assume that respondents would be discouraged from answering the peer survey honestly if they had to account publicly for their opinions. We also don't know who actually fills out a survey: We know only, for example, whether the dean's survey has been returned.

neil of CT 8:58PM April 06, 2008