Grad Rankings: Share Your Feelings
Reader Comments
database of universities, colleges http://www.orionicon.com
database of universities, colleges
http://www.orionicon.com
database of universities, colleges http://www.orionicon.com
database of universities, colleges
http://www.orionicon.com
Medical School Rankings - Self Reported Data Validation?
Bob,
I was looking at the Medical School rankings and the associated trends in full-time faculty counts as reported/implied in the fac/student ratio and NIH/faculty components of the Research Rankings of Medical Schools.
Was wondering what sort of internal checks are done on the self-reported data. For example, did you investigate why Yale's full-time faculty count dropped by nearly 500 between the 2006 and 2007 rankings?
Ranking Year | Yale's Full time Faculty Count
2001 1342
2002 1389
2003 1436
2004 1420
2005 1470
2006 1503
2007 1014
2008 1024
2009 1063
Coincidentally 2006 was the first year the NIH/faculty metric was introduced, where a school with a smaller faculty count would benefit, as Yale did in their rankings trend (11 in 2006 to 9 in 2007 to 8 in 2008).
What sort of checks are in place for this sort of reporting variablity from year to year?
Regards,
Matt
N/A for employed at graduation
I was looking forward to the answer to the following questions:
"What number is used for the law school employed at graduation number if a school does not report that data (i.e., N/A appears)? For example, if a school has a 95% employed 9 month number, what number would be used for the employed at grad? Are there any instances where it would be more beneficial for a school to choose not to report its at grad numbers?"
The reply to that post does not seem to respond to these questions.
Bob Morse responds: U.S. News estimates a value for ranking purposes only, if a school doesn't report their "at grad" employment rate. That value is approximately 30 pecentage points less than that school's 9 month rate. So, for the 95% 9-month school the value would be approximately 95-30=65, or 65%.
Mr. Morse,
Thank you for your response. I will try to make this the last post, as I think the point has gotten across, and I'm simply belaboring it.
I can't say how many schools did what Pepperdine did, but that should not take away from the fact that we know Pepperdine did it! Its recent rise is more than highly suspicious - I think now the more you dig, the more that you will find that it was the result of conscious action.
Notice how they went back to "verify" those studying for the bar full time, and not seeking work, the categories that hurt them under the new methodology, but did they also go back to "verify" those who originally reported being employed, and did that result in any changes to the contrary? I bet not.
Schools should not have been put on notice last year of the coming methodology change, because it gave them opportunity to do what Pepperdine did. I guarantee you if the change hadn't been made public, they'd have submitted the original numbers they emailed me, because under the old methodology, those numbers would have been great.
Maybe it would help to do away with the at grad component, a number that is highly manipulable and not reported to the ABA, and shift its weight to Bar pass rate, a very useful indicator that is not at all subject to manipulation.
Mr. Morse, I thank you for your ear, and hope that now, you can see the depths of the deception being perpetrated in order to artificially enhance the rankings.
Pepperdine Redux
Mr. Morse:
A few follow up items:
What number is used for the law school employed at graduation number if a school does not report that data (i.e., N/A appears)? For example, if a school has a 95% employed 9 month number, what number would be used for the employed at grad? Are there any instances where it would be more beneficial for a school to choose not to report its at grad numbers?
Example, Pepperdine has historically reported its employed at grad number up through the 2007 rankings, where it reported an employed at grad rate of 57.5%. Its rank was 87.
Curiously, and suspiciously, in the subsequent year, it did not report that figure as it had always done in the past, and its rank shot up 21 spots to 66. The other input variables for Pepperdine did not change much, in fact, its Judge/lawyer rep score actually went down from 3.1 in 2007 to 2.9 in 2008.
Could it be that Pepperdine consciously chose to stop reporting the employed at graduation number because that variable hurt them, and the alternative of not reporting yielded a more favorable result?
Also, since when do schools get a "do over" on their nine month reporting numbers? I don't think staff turnover warrants a redo on employment surveys that had already been completed. The email from Pepperdine essentially admits that they called students and reconstructed the numbers, which had already been compiled by March.
Isn't it curious that that exercise resulted in a far more beneficial result, and magically, corresponded with the change in methodology? What did those phone calls to students from Pepperdine career services sound like, I wonder? "hi this is career services...you already completed a survey telling us you were unemployed, but did you really mean that? No, ok, great, we'll mark you as employed"
Mr. Morse, could it be that Pepperdine realized that, as a result of the change in methodology, the career stats it had compiled under the old methodology would be harmful, and so it embarked on a mission to re-document its numbers after the fact in a way that would be beneficial?
It is unfortunate that schools will go to these lengths, but as you can see, nothing is beyond them....I'm almost inclined to write a letter to the ABA asking them to investigate this.
Bob Morse responds: You make a good point about Pepperdine's data. But, there is one big unknown. USNWR does not know how many other law, if any, schools have done the same thing which is to change/update their jobs data with NALP and/or the ABA after it has been filed.
re: Leiter's way is not the way to go
Both the previous comment and the response to it are difficult for me to make sense of. Anon suggests that Brian Leiter is elitist and has different interests than most students considering law school. But Leiter's recommendations for the rankings aim to make them a more reliable measure of school quality than they are today, and if anything would make it easier for less wealthy schools to perform well on the rankings. For example, Leiter suggests getting rid of per capita expenditures as one of the factors in the rankings. The schools who stand to gain from such an adjustment in the methodology are those which have less money per student to spend. Leiter also suggests ways to make the reputational survey and the other factors in the U.S. News rankings more accurate. What is elitist about that?
With all due respect, Mr. Morse, your response is equally puzzling to me. Are you saying that you refuse to accept Prof. Leiter's suggestions because you hold yourself to a lower standard than he does? Everyone agrees with you that the U.S. News rankings do not purport to be social science. But that does not seem like a good reason not to accept some thoughtful suggestions for improving the rankings.
Bob Morse responds. Prof. Leiter http://leiterlawschool.typepad.com/leiter/2008/03/an-open-lette-1.html suggests (my reading of his letter) that we change the academic reputation survey to make it much more academic focused (involving an very indepth additional data collection to establish detailed law school profiles that), drop expenditures per student and drop at % of grads employed data because both are self reported (of course they are reported to the ABA). Many argue simply that resources do matter a great deal and tuition dependent schools offer far less to students than those law schools that aren't cash cows for their parent unversities. He does suggest we add a federal clerkship indicator and a measure of grads going into major firms. Those are big changes. My point is that many people are now making suggestions to USNEWS and we have to study them all carefully. We hope to get organzied feedback from the law school deans, too. We need to take into account these many different views before we decide what, if any, changes we should and are able to implement.
Rankings Article
Mr. Morse,
All of the concerns addressed by Prof. Leiter and others were ably addressed in this law review article (you may have read this, but if not, I think you would find it to be useful and important).
http://papers.ssrn.com/sol3/papers.cfm?abstract_id=937017
Bob Morse responds: I agree this paper "Understanding the U.S. News Law School Rankings" by THEODORE P. SETO @ Loyola Law School Los Angeles
-------------------------------------------------------------------------------- is probably the most complete analysis written about the USNEWS rankings. I have read it and spoke to Seto about it.
Leiter's way is not the way to go
I wanted to respond to an earlier post above. If you adopted Professor Brian Leiter's rankings you would be caving in. Your rankings would lose their power to make law schools accountable. While I admire Leiter for his focus on elitist scholars, his interests are *not* the interests of 95% of the students considering law school. Really, you are the only institution with us, the undergraduates, in mind. Do not accommodate yourself to what the tenured law professors want.
Bob Morse responds: USNEWS is fully aware (implied by my ealier comment) that if we make Leiter's changes our rankings will not be what they are today. They are geared to consumers i.e. undergrads, law students and the general public. USNEWS is not claiming we are doing rankings that would meet social science research standards and pass peer review.
Law School Rankings
Mr. Morse
I have two related concerns with the current ranking methodology for law schools:
1. Employment data -- Including employment data would seem to influences the outcome of future employment prospects. I'd be interested in seeing a statistical study about how the rankings influence hiring and recruitment decisions. Most large firms limit their recruitment to higher-ranked schools, and a hiring partner's eyes would likely gravitate towards a top-10 school's graduate compared to a 3rd tier school's graduate. If employment data is included in weighting, then it could influence the employment data for the following year's rankings.
2. Hiring partner and judges surveys - You stated that you mainly asking hiring partners and judges-those that hire new minted J.D.s. Similar to employment data, these surveys become perpetually biased if there is no standardization of what the hiring partners and judges know about each school. Have they met the professors or school administration? Do they know the current bar pass rates of the schools? Do they all know or currently work with students at the schools they rank? Are the surveys only done by those local to the schools? Because someone knows the name of a school and it has "a good reputation" does not necessarily mean that a schol with less name recognition isn't worthy of "a good reputation."
The overarching issue for including both of these measures of quality is that they strongly influence *future* data used to calculate rankings. This is a case of the statistical study having influence upon those being observed (i.e. the survey takers). It's akin to saying to the survey-takers, "Here are the best schools from last year. After seeing that, rank the best schools for this year." A more scientific and unbiased methodology would attempt to have minimal influence.
A separate concern that I have is that your weighting only 0.02 for bar exam passage rate. The relative rankings for schools are absurd because of the low weighting for this measure of quality. The rankings purportedly measure the quality of students entering and graduating the school with LSAT, GPA, bar pass, and employment data. The quality of the students within and graduating from the school should give the greatest measure of the schools' quality. The bar exam is purported to assure the competency of lawyers. Thus having a higher pass rate presumably assures higher overall competency of students becoming lawyers.
I graduated from the University of San Francisco. USF is *Third Tier* this year with 85% pass rate for CA July 2007 exam. I know July 2007 exams may not reflect in the rankings until next year, but USF's third tier ranking is extremely different from following top-100 schools (including one nearly 100 rankings higher with a lower bar passage rate)?
It had a bar pass rate relative to:
Boalt Hall (ranked #6 with 82%)
University of California –Hastings College of The Law (#38 with 81%)
University of California - Davis (#44 with 84%)
Pepperdine University School of Law (#59 with 78%)
Loyola Law School-Los Angeles (#63 with 78%)
Santa Clara University School of Law (#77 with 76%)
University of San Diego School of Law (#82 with 80%)
McGeorge School of Law (ranked #95 with 73%)
I look forward to your response. Thank you!
Bob Morse responds: Your post is long and thoughtful. Difficult to respond to in brief. You are correct that we should consider boosting the weight of the bar passage rate since it's a good outcomes measure. U.S. News has not done any statisitical studies about the impact of our rankings on hiring and recruiting, I know that some law professors and others have. Maybe they will comment on my blog. We do need to study ways to improve the lawyer/judge survey.
