Grad Rankings: Share Your Feelings

Reader Comments
Back to blog

After reading you blog, Your blog is very useful for me .I bookmarked your blog! I trust you will behave better from now on; I hope she understands that she cannot exepct a raise.

free trial of AL 3:18AM February 14, 2010

database of universities, colleges
http://www.orionicon.com

ahmed of IN 3:11AM October 15, 2009

database of universities, colleges
http://www.orionicon.com

ahmed of KY 3:05AM October 15, 2009

Bob,
I was looking at the Medical School rankings and the associated trends in full-time faculty counts as reported/implied in the fac/student ratio and NIH/faculty components of the Research Rankings of Medical Schools.
Was wondering what sort of internal checks are done on the self-reported data. For example, did you investigate why Yale's full-time faculty count dropped by nearly 500 between the 2006 and 2007 rankings?
Ranking Year | Yale's Full time Faculty Count
2001 1342
2002 1389
2003 1436
2004 1420
2005 1470
2006 1503
2007 1014
2008 1024
2009 1063
Coincidentally 2006 was the first year the NIH/faculty metric was introduced, where a school with a smaller faculty count would benefit, as Yale did in their rankings trend (11 in 2006 to 9 in 2007 to 8 in 2008).
What sort of checks are in place for this sort of reporting variablity from year to year?
Regards,
Matt

Matt of MI 11:03AM June 30, 2008

I was looking forward to the answer to the following questions:
"What number is used for the law school employed at graduation number if a school does not report that data (i.e., N/A appears)? For example, if a school has a 95% employed 9 month number, what number would be used for the employed at grad? Are there any instances where it would be more beneficial for a school to choose not to report its at grad numbers?"
The reply to that post does not seem to respond to these questions.
Bob Morse responds: U.S. News estimates a value for ranking purposes only, if a school doesn't report their "at grad" employment rate. That value is approximately 30 pecentage points less than that school's 9 month rate. So, for the 95% 9-month school the value would be approximately 95-30=65, or 65%.

m of FL 12:21PM May 09, 2008

Mr. Morse,
Thank you for your response. I will try to make this the last post, as I think the point has gotten across, and I'm simply belaboring it.
I can't say how many schools did what Pepperdine did, but that should not take away from the fact that we know Pepperdine did it! Its recent rise is more than highly suspicious - I think now the more you dig, the more that you will find that it was the result of conscious action.
Notice how they went back to "verify" those studying for the bar full time, and not seeking work, the categories that hurt them under the new methodology, but did they also go back to "verify" those who originally reported being employed, and did that result in any changes to the contrary? I bet not.
Schools should not have been put on notice last year of the coming methodology change, because it gave them opportunity to do what Pepperdine did. I guarantee you if the change hadn't been made public, they'd have submitted the original numbers they emailed me, because under the old methodology, those numbers would have been great.
Maybe it would help to do away with the at grad component, a number that is highly manipulable and not reported to the ABA, and shift its weight to Bar pass rate, a very useful indicator that is not at all subject to manipulation.
Mr. Morse, I thank you for your ear, and hope that now, you can see the depths of the deception being perpetrated in order to artificially enhance the rankings.

Ken of NJ 8:00PM May 02, 2008

Mr. Morse:
A few follow up items:
What number is used for the law school employed at graduation number if a school does not report that data (i.e., N/A appears)? For example, if a school has a 95% employed 9 month number, what number would be used for the employed at grad? Are there any instances where it would be more beneficial for a school to choose not to report its at grad numbers?
Example, Pepperdine has historically reported its employed at grad number up through the 2007 rankings, where it reported an employed at grad rate of 57.5%. Its rank was 87.
Curiously, and suspiciously, in the subsequent year, it did not report that figure as it had always done in the past, and its rank shot up 21 spots to 66. The other input variables for Pepperdine did not change much, in fact, its Judge/lawyer rep score actually went down from 3.1 in 2007 to 2.9 in 2008.
Could it be that Pepperdine consciously chose to stop reporting the employed at graduation number because that variable hurt them, and the alternative of not reporting yielded a more favorable result?
Also, since when do schools get a "do over" on their nine month reporting numbers? I don't think staff turnover warrants a redo on employment surveys that had already been completed. The email from Pepperdine essentially admits that they called students and reconstructed the numbers, which had already been compiled by March.
Isn't it curious that that exercise resulted in a far more beneficial result, and magically, corresponded with the change in methodology? What did those phone calls to students from Pepperdine career services sound like, I wonder? "hi this is career services...you already completed a survey telling us you were unemployed, but did you really mean that? No, ok, great, we'll mark you as employed"
Mr. Morse, could it be that Pepperdine realized that, as a result of the change in methodology, the career stats it had compiled under the old methodology would be harmful, and so it embarked on a mission to re-document its numbers after the fact in a way that would be beneficial?
It is unfortunate that schools will go to these lengths, but as you can see, nothing is beyond them....I'm almost inclined to write a letter to the ABA asking them to investigate this.
Bob Morse responds: You make a good point about Pepperdine's data. But, there is one big unknown. USNWR does not know how many other law, if any, schools have done the same thing which is to change/update their jobs data with NALP and/or the ABA after it has been filed.

Ken of NJ 3:25PM May 01, 2008

Both the previous comment and the response to it are difficult for me to make sense of. Anon suggests that Brian Leiter is elitist and has different interests than most students considering law school. But Leiter's recommendations for the rankings aim to make them a more reliable measure of school quality than they are today, and if anything would make it easier for less wealthy schools to perform well on the rankings. For example, Leiter suggests getting rid of per capita expenditures as one of the factors in the rankings. The schools who stand to gain from such an adjustment in the methodology are those which have less money per student to spend. Leiter also suggests ways to make the reputational survey and the other factors in the U.S. News rankings more accurate. What is elitist about that?
With all due respect, Mr. Morse, your response is equally puzzling to me. Are you saying that you refuse to accept Prof. Leiter's suggestions because you hold yourself to a lower standard than he does? Everyone agrees with you that the U.S. News rankings do not purport to be social science. But that does not seem like a good reason not to accept some thoughtful suggestions for improving the rankings.
Bob Morse responds. Prof. Leiter http://leiterlawschool.typepad.com/leiter/2008/03/an-open-lette-1.html suggests (my reading of his letter) that we change the academic reputation survey to make it much more academic focused (involving an very indepth additional data collection to establish detailed law school profiles that), drop expenditures per student and drop at % of grads employed data because both are self reported (of course they are reported to the ABA). Many argue simply that resources do matter a great deal and tuition dependent schools offer far less to students than those law schools that aren't cash cows for their parent unversities. He does suggest we add a federal clerkship indicator and a measure of grads going into major firms. Those are big changes. My point is that many people are now making suggestions to USNEWS and we have to study them all carefully. We hope to get organzied feedback from the law school deans, too. We need to take into account these many different views before we decide what, if any, changes we should and are able to implement.

JIK of MD 9:33PM April 12, 2008

Mr. Morse,
All of the concerns addressed by Prof. Leiter and others were ably addressed in this law review article (you may have read this, but if not, I think you would find it to be useful and important).
http://papers.ssrn.com/sol3/papers.cfm?abstract_id=937017
Bob Morse responds: I agree this paper "Understanding the U.S. News Law School Rankings" by THEODORE P. SETO @ Loyola Law School Los Angeles
-------------------------------------------------------------------------------- is probably the most complete analysis written about the USNEWS rankings. I have read it and spoke to Seto about it.

bill of CA 4:43PM April 12, 2008

I wanted to respond to an earlier post above. If you adopted Professor Brian Leiter's rankings you would be caving in. Your rankings would lose their power to make law schools accountable. While I admire Leiter for his focus on elitist scholars, his interests are *not* the interests of 95% of the students considering law school. Really, you are the only institution with us, the undergraduates, in mind. Do not accommodate yourself to what the tenured law professors want.
Bob Morse responds: USNEWS is fully aware (implied by my ealier comment) that if we make Leiter's changes our rankings will not be what they are today. They are geared to consumers i.e. undergrads, law students and the general public. USNEWS is not claiming we are doing rankings that would meet social science research standards and pass peer review.

anon of NE 11:06PM April 11, 2008