The Wait for the National Research Council Rankings Continues

Reader Comments
Back to blog

I think that the NRC report is producing something fundamentally different from U.S. News and World Report’s rankings. It is giving an in-depth qualitative evaluation of graduate programs and doing this with respect to graduate students and professor opinion. In contrast, U.S. News and World Report is taking figures and producing a numerical output that may not correspond with the actual value a graduate program. Within many disciplines, there may be more unperceived characteristics of a graduate program of interest to students than U.S. News and World rankings may be sensitive too. This may be why NRC rankings showed Harvard to be somewhere lower than what its endowment would have one believe, many of its research programmes are stagnant and not influential, a characteristic U.S. News staff may be unqualified to assess.

Ricardo Lezama of CA 6:13AM December 08, 2010

U.S. News should expand the fields that you rank in your Graduate Schools report, so that you would in effect do a yearly comprehensive NRC-type ranking. And consider the methodology NRC used to see whether you can improve yours.

Greg Mantell of CA 8:25PM October 25, 2010

The NRC was to have had its data available in 2007, but they kept putting it off, revising, and apparently making up their methods on the fly. Information can become so sophisticated and outdated as to be irrelevant. I suspect that in trying to control how their report will be used by other ranking agencies, they've exceeded their ability to produce an objective survey. Conversion of their 1995 data into mathematically significant output showed Harvard University ranking in the twenties.

james andrew of FL 1:54PM August 03, 2010

i would be greatly thankful to you if you provide me all the information related to the above given college the locality, the rankings, the employment opportunities the availability of part time jobs for international students who would like to do MS in electrical engineering in Texas A&M university ,Kingsville

harish of TX 9:01AM December 29, 2009

This is a more vivid example of the underrating of public universities in comparison to private by US News Undergraduate faculty quality of academic reputation scores(in parenthesis) as opposed the the number of top ten progrsms by the NRC 1995 reports.
School programs in UW News Undergrad
Top Ten(NRC) academic reputation score
Berkeley 38 4.8
Stanford 32 4.9
Harvard 28 4.9
Princeton 22 4.9
MIT 20 4.9
Yale 19 4.9
Cornell 19 4.6
Chicago 18 4.7
Penn 15 4.5
UC San Diego 14 3.8
Wisconsin 14 4.1
Columbia 14 4.7
Michigan 14 4.5
UCLA 13 4.2
Univ Washington 11 4.0
Illinois 10 4.1
Johns Hopkins 9 4.7
Duke 8 4.6
Texas 7 4.1
Minnesota 5 3.9
Georgetown 0 4.0
Washington Univ 0 4.1
Notre Dame 0 4.0
Vanderbilt 0 4.0

David White of OH 2:46PM August 11, 2009

WHILE THE US NEWS GRADUATE SCORES AND THE NRC GRADUATE SCORES ARE HIGHLY CORRELATED (0.92),AS THEY SHOUULD BE, THE US NEWS GRADUATE SCORES AND THE US NEWS UNDERGRADUATE FACULTY QUALITY OR ACADEMIC REPUTATION SCORES ARE NOT CORRELATE (0.51), NOR ARE THEY CORRELATE WITH THE NRC SCORES (0.52), AND THEY SHOULD BE. THIS PRESENTS A PROBLEM FOR US NEWS AS IT HIGHLIGHTS THE MAJOR FLAW IN METHODOLOGY IN THE US NEWS UNDERGRADUATE RANKINGS.
THE SOLUTION IS SIMPLE...INSTEAD OF POLLING DEANS OF ADMISSION OR CHANCELLORS FOR ACADEMIC REPUTATIONAL RATINGS (THEY KNOW VERY LITTLE ABOUT THIS), POLL HEADS OF INDIVIDUAL DEPARTMENTS:BIOLOGY, CHEMISTRY, ENGLISH, PHYSICS, ECONOMICS, MATHEMATICS, PSYCHOLOGY, SOCIOLOGY, COMPUTER SCIENCE, ETC, AGGREGATE AND TRANSFORM THIS DATA AND USE IT AS THE FACULTY QUALITY SCORE. YOU WILL FIND THAT YOU GET A VERY DIFFERENT AND MORE ACCURATE RESULT THAN WHAT YOU HAVE NOW, WHICH IS AN UNDERRATING OF THE ACADEMIC REPUTATION VARIABLE FOR PUBLIC SCHOOLS AND AN OVERRATING OF THIS VARIABLE FOR PRIVATE SCHOOLS IN THE US NEWS UNDERGRADUATE RANKINGS.
PEOPLE CAN ARGUE ABOUT THE WEIGHT GIVEN TO REST OF THE VARIABLES IN THE US NEWS UNDERGRADUATE RANKINGS, ALL OF WHICH ARE QUANTITATIVE, BUT INACCURATE MEASUREMENTS OF A SCHOOLS REPUTATION IS MISLEADING AT THE VERY LEAST. AS AN EXAMPLE, A SCHOOL LIKE UC SAN DIEGO WHICH IS THE MOST UNDERRATED IN REPUTATION SHOULD HAVE A SCORE OF 4.6 ACCORDING TO THE NRC SCORES AND WHICH NOW IS GIVEN A SCORE OF 3.8, IS A GREAT SCHOOL AND READERS SHOULD KNOW THAT WHEN THEY ARE CONSIDERING SPENDING $140000 FOR A 4 YEAR SCHOOL. THE CORRECT SCORE WOULD MOVE UC SAN DIEGO UP ALMOST 10 PLACES IN THE OVERALL RANKINGS. AS YOU KNOW, SMALL MOVEMENTS IN RANKINGS CAN HAVE A HUGE IMPACT ON THE DECISION OF APPLICANTS AND THE NUMBER OF APPLICANTS FOR ANY GIVEN SCHOOL. IN ADDITION, SCHOOLS LIKE GEORGETOWN, JOHNS HOPKINS, DUKE OR WASHINGTON UNIVERSITY, HAVE ACADEMIC REPUTATIONS WHICH ARE HIGHLY OVERRATED ACCORDING TO THE DATA FROM THE US NEWS UNDERGRADUATE FACULTY QUALITY AND WHEN IT IS COMPARED TO THE NRC(NATIONAL RESEARCH COUNCIL) DATA.
IF YOU LOOK THE PREVIOUS FACULTY QUALITY RATINGS GOING BACK TO 1925, YOU WILL FIND CONSISTENT RANKINGS THROUGOUT THE LAST 85 YEARS, EVEN IF YOU USE THE CURRENT US NEWS AGGREGATED GRADUATE SCORES AS A COMPARISON. BUT, IF THE FACULTY QUALITY OR ACADEMIC REPUTATION VARIABLE SCORES FROM THE US NEWS UNDERGRADUATE RANKINGS ARE USED TO COMPARE WITH THE LAST 85 YEAR HISTORY, THERE IS ALMOST NO CORRELATION WITH THE HISTORICAL DATA.
SINCE THIS VARIABLE ACCOUNTS FOR 25% OF THE TOTAL AND IS THE ONLY QUALITATIVE VARIABLE, THE METHODOLOGY MUST BE ACCURATE AND WHICH, AT PRESENT, IT IS NOT.

David White of OH 8:29AM August 11, 2009

RECENTLY i DID A STUDY ON THE US NEWS UNDERGRADUATE SCORES VS. THE NRC SCORES AND FOUND THAT WHILE THE CORRELATION BETWEEN THE US NEWS GRADUATE SCORES CORRELATE VERY HIGHLY (0.91) WITH THE NRC SCORES, THE US NEWS UNDERGRADUATE FACULTY QUALITY SCORES CORRELATE VERY POORLY(0.52). THE BIGGEST DISCREPENCY IS WITH PUBLIC SCHOOLS WHICH ARE EXTREMELY UNDERRATED (84%)IN THE US NEWS UNDERGRADUATE FACULTY QUALITY VARIABLES AND PRIVATES SCHOOLS ARE ALMOST ALL OVERRATED(75%).
WHEN YOU USE THE NRC SCORES AND SCALE THEM ON A 1-5 SCALE, THE DIFFERENCES BECOME VERY OBVIOUS. FOR INSTANCE, CURRENTLY UC SAN DIEGO GETS A 3.8 ON THE US NEWS UNDERGRADUATE FACULTY QUALITY SCORE BUT GETS A 4.6 WHEN YOU USE A TRANSFORMED NRC SCORE..SOME OTHER INTERESTING RESULTS USING TRANSFORMED NRC SCORES(CURRENT US NEWS UNDERGRADUATE FACULTY QUALITY SCORES IN PARENTHESIS):
UC SAN DIEGO 4.6 (3.8) MOST UNDERRATED FACULTY IN US NEWS UNDERGRAD
WSCONSIN 4.5(4.1)
UCLA 4.5(4.2)
WASHINGTON UNIV. 3.6 (4.1)
UNIV WASHINGTON 4.3(3.9)
MINNESOTA 4.2 (3.8)
NORTH CAROLINA STATE 3.9(3.2)
GEORGETOWN 3.5(4.0)
NOTRE DAME 3.5(4.0)
YALE 4.7(4.9)
VIRGINIA 3.9(4.3) ONE OF THE FEW OVERRATED PUBLIC SCHOOLS
DUKE 4.2(4.6)
JOHNS HOPKINS 4.1(4.6)THE MOST OVERRATED UNIVERSITY IN US NEWS UNDERGRAD
UNIV ROCHESTER 4.1 (3.6) UNDERRATED PRIVATE SCHOOLS
CAL TECH 4.4 (4.7)

DAVID WHITE of OH 12:37PM August 08, 2009

You can look down your nose at the US News rankings, but many people would be interested in examining the degree of association between the two rankings. The platitude "garbage in/garbage out" is just not appropriate in this case. When people self select and call into a tv or radio station to vote on some issue, that is an excellent example of garbage data and the results are statistically meaningless. To include the US News rankings as a covariate in a regression model allows the data to speak for themselves and tell us whether the rankings are predictive. Suppose the US News rankings are actually obtained via a Ouija board- in such a situation over the vast majority of fields one would expect to find no association with the "gold standard". (However, due to chance variation (under the so-called null hypothesis) over many fields some correlations may be statistically significant and this needs to be corrected for.) I fail to see why one should be closed minded and ignore the US News rankings- let the data tell us whether they have any predictive power. Regarding variability of rankings over time, I took a quick look at the US News results for 2006 and for 2008 in statistics. While not an indepth study of variation, Stanford and Berkeley stayed #1 and #2, while Harvard and UNC switched places. UW-Madison and Cornell fell off the top ten list to be replaced by Iowa State and Duke. Univ of Chicago's rank remained constant and U of Washington moved up one notch. All in all not very different than what I would have predicted and not unrealistic. I expect, as one moves down the list, the variance of the rank increases (probably to the point of being random at some point) and that should be reflected in the width of the interval provided by the NRC analysis.

statman 3:52PM July 14, 2009

Anyone who has been to higher education knows that academia changes extremely slowly. So any ranking that alters the order of colleges dramatically differently from year to year is not worthy of studying by anyone, let alone the National Research Council. It makes for great drama (thus more profitable for your magazine) that Stanford, e.g., is #3 in one year but plummeted to #15 the following year, and Harvard is #1 this year but fell out of the top 10 altogether the next. But such drama is highly suspect among educators, and the academe (including the NRC) does operate on a different truth plane than that used by commercial businesspeople. You can devise a valid statistical machine for analysis, but if you don't take the time to examine your data, then it's just garbage in and garbage out.

TS of CA 11:57PM July 13, 2009

The beauty of multiple regression is that it does not matter whether the rankings are randomly generated by a computer, by a three year-old playing with blocks or by the most thoughtful and error free methodology available. In a regression analysis the rankings would be included as another covariate in the regression model and we let the data inform us as to the predictability of the approach. The US News rankings (whether or not the methodology is understood) are available and well known to the public. I maintain that it would be an interesting empirical exercise (and an obvious one) to examine the association between the two approaches for each field. The
remark "According to the U.S. News Editor Brian Kelly, the magazine's college ranking formula is 'proprietary',so there is no way NRC or anyone can compare it" indicates a lack of understanding of what multiple regression does and how it works.
To do the empirical analysis you don't need the ranking formula-just the rankings.

statman 10:41PM July 13, 2009