The Wait for the National Research Council Rankings Continues

Top Ten programs vs. US News Undergrad academic reputation score

This is a more vivid example of the underrating of public universities in comparison to private by US News Undergraduate faculty quality of academic reputation scores(in parenthesis) as opposed the the number of top ten progrsms by the NRC 1995 reports.

School programs in UW News Undergrad

Top Ten(NRC) academic reputation score

Berkeley 38 4.8

Stanford 32 4.9

Harvard 28 4.9

Princeton 22 4.9

MIT 20 4.9

Yale 19 4.9

Cornell 19 4.6

Chicago 18 4.7

Penn 15 4.5

UC San Diego 14 3.8

Wisconsin 14 4.1

Columbia 14 4.7

Michigan 14 4.5

UCLA 13 4.2

Univ Washington 11 4.0

Illinois 10 4.1

Johns Hopkins 9 4.7

Duke 8 4.6

Texas 7 4.1

Minnesota 5 3.9

Georgetown 0 4.0

Washington Univ 0 4.1

Notre Dame 0 4.0

Vanderbilt 0 4.0

REVISE ACADEMIC REPUTATION METHODOLOGY

WHILE THE US NEWS GRADUATE SCORES AND THE NRC GRADUATE SCORES ARE HIGHLY CORRELATED (0.92),AS THEY SHOUULD BE, THE US NEWS GRADUATE SCORES AND THE US NEWS UNDERGRADUATE FACULTY QUALITY OR ACADEMIC REPUTATION SCORES ARE NOT CORRELATE (0.51), NOR ARE THEY CORRELATE WITH THE NRC SCORES (0.52), AND THEY SHOULD BE. THIS PRESENTS A PROBLEM FOR US NEWS AS IT HIGHLIGHTS THE MAJOR FLAW IN METHODOLOGY IN THE US NEWS UNDERGRADUATE RANKINGS.

THE SOLUTION IS SIMPLE...INSTEAD OF POLLING DEANS OF ADMISSION OR CHANCELLORS FOR ACADEMIC REPUTATIONAL RATINGS (THEY KNOW VERY LITTLE ABOUT THIS), POLL HEADS OF INDIVIDUAL DEPARTMENTS:BIOLOGY, CHEMISTRY, ENGLISH, PHYSICS, ECONOMICS, MATHEMATICS, PSYCHOLOGY, SOCIOLOGY, COMPUTER SCIENCE, ETC, AGGREGATE AND TRANSFORM THIS DATA AND USE IT AS THE FACULTY QUALITY SCORE. YOU WILL FIND THAT YOU GET A VERY DIFFERENT AND MORE ACCURATE RESULT THAN WHAT YOU HAVE NOW, WHICH IS AN UNDERRATING OF THE ACADEMIC REPUTATION VARIABLE FOR PUBLIC SCHOOLS AND AN OVERRATING OF THIS VARIABLE FOR PRIVATE SCHOOLS IN THE US NEWS UNDERGRADUATE RANKINGS.

PEOPLE CAN ARGUE ABOUT THE WEIGHT GIVEN TO REST OF THE VARIABLES IN THE US NEWS UNDERGRADUATE RANKINGS, ALL OF WHICH ARE QUANTITATIVE, BUT INACCURATE MEASUREMENTS OF A SCHOOLS REPUTATION IS MISLEADING AT THE VERY LEAST. AS AN EXAMPLE, A SCHOOL LIKE UC SAN DIEGO WHICH IS THE MOST UNDERRATED IN REPUTATION SHOULD HAVE A SCORE OF 4.6 ACCORDING TO THE NRC SCORES AND WHICH NOW IS GIVEN A SCORE OF 3.8, IS A GREAT SCHOOL AND READERS SHOULD KNOW THAT WHEN THEY ARE CONSIDERING SPENDING $140000 FOR A 4 YEAR SCHOOL. THE CORRECT SCORE WOULD MOVE UC SAN DIEGO UP ALMOST 10 PLACES IN THE OVERALL RANKINGS. AS YOU KNOW, SMALL MOVEMENTS IN RANKINGS CAN HAVE A HUGE IMPACT ON THE DECISION OF APPLICANTS AND THE NUMBER OF APPLICANTS FOR ANY GIVEN SCHOOL. IN ADDITION, SCHOOLS LIKE GEORGETOWN, JOHNS HOPKINS, DUKE OR WASHINGTON UNIVERSITY, HAVE ACADEMIC REPUTATIONS WHICH ARE HIGHLY OVERRATED ACCORDING TO THE DATA FROM THE US NEWS UNDERGRADUATE FACULTY QUALITY AND WHEN IT IS COMPARED TO THE NRC(NATIONAL RESEARCH COUNCIL) DATA.

IF YOU LOOK THE PREVIOUS FACULTY QUALITY RATINGS GOING BACK TO 1925, YOU WILL FIND CONSISTENT RANKINGS THROUGOUT THE LAST 85 YEARS, EVEN IF YOU USE THE CURRENT US NEWS AGGREGATED GRADUATE SCORES AS A COMPARISON. BUT, IF THE FACULTY QUALITY OR ACADEMIC REPUTATION VARIABLE SCORES FROM THE US NEWS UNDERGRADUATE RANKINGS ARE USED TO COMPARE WITH THE LAST 85 YEAR HISTORY, THERE IS ALMOST NO CORRELATION WITH THE HISTORICAL DATA.

SINCE THIS VARIABLE ACCOUNTS FOR 25% OF THE TOTAL AND IS THE ONLY QUALITATIVE VARIABLE, THE METHODOLOGY MUST BE ACCURATE AND WHICH, AT PRESENT, IT IS NOT.

US NEWS UNDERGRAD FACULTY QUALITY SCORES VS NRC SCORES

RECENTLY i DID A STUDY ON THE US NEWS UNDERGRADUATE SCORES VS. THE NRC SCORES AND FOUND THAT WHILE THE CORRELATION BETWEEN THE US NEWS GRADUATE SCORES CORRELATE VERY HIGHLY (0.91) WITH THE NRC SCORES, THE US NEWS UNDERGRADUATE FACULTY QUALITY SCORES CORRELATE VERY POORLY(0.52). THE BIGGEST DISCREPENCY IS WITH PUBLIC SCHOOLS WHICH ARE EXTREMELY UNDERRATED (84%)IN THE US NEWS UNDERGRADUATE FACULTY QUALITY VARIABLES AND PRIVATES SCHOOLS ARE ALMOST ALL OVERRATED(75%).

WHEN YOU USE THE NRC SCORES AND SCALE THEM ON A 1-5 SCALE, THE DIFFERENCES BECOME VERY OBVIOUS. FOR INSTANCE, CURRENTLY UC SAN DIEGO GETS A 3.8 ON THE US NEWS UNDERGRADUATE FACULTY QUALITY SCORE BUT GETS A 4.6 WHEN YOU USE A TRANSFORMED NRC SCORE..SOME OTHER INTERESTING RESULTS USING TRANSFORMED NRC SCORES(CURRENT US NEWS UNDERGRADUATE FACULTY QUALITY SCORES IN PARENTHESIS):

UC SAN DIEGO 4.6 (3.8) MOST UNDERRATED FACULTY IN US NEWS UNDERGRAD

WSCONSIN 4.5(4.1)

UCLA 4.5(4.2)

WASHINGTON UNIV. 3.6 (4.1)

UNIV WASHINGTON 4.3(3.9)

MINNESOTA 4.2 (3.8)

NORTH CAROLINA STATE 3.9(3.2)

GEORGETOWN 3.5(4.0)

NOTRE DAME 3.5(4.0)

YALE 4.7(4.9)

VIRGINIA 3.9(4.3) ONE OF THE FEW OVERRATED PUBLIC SCHOOLS

DUKE 4.2(4.6)

JOHNS HOPKINS 4.1(4.6)THE MOST OVERRATED UNIVERSITY IN US NEWS UNDERGRAD

UNIV ROCHESTER 4.1 (3.6) UNDERRATED PRIVATE SCHOOLS

CAL TECH 4.4 (4.7)

GIGO

You can look down your nose at the US News rankings, but many people would be interested in examining the degree of association between the two rankings. The platitude "garbage in/garbage out" is just not appropriate in this case. When people self select and call into a tv or radio station to vote on some issue, that is an excellent example of garbage data and the results are statistically meaningless. To include the US News rankings as a covariate in a regression model allows the data to speak for themselves and tell us whether the rankings are predictive. Suppose the US News rankings are actually obtained via a Ouija board- in such a situation over the vast majority of fields one would expect to find no association with the "gold standard". (However, due to chance variation (under the so-called null hypothesis) over many fields some correlations may be statistically significant and this needs to be corrected for.) I fail to see why one should be closed minded and ignore the US News rankings- let the data tell us whether they have any predictive power. Regarding variability of rankings over time, I took a quick look at the US News results for 2006 and for 2008 in statistics. While not an indepth study of variation, Stanford and Berkeley stayed #1 and #2, while Harvard and UNC switched places. UW-Madison and Cornell fell off the top ten list to be replaced by Iowa State and Duke. Univ of Chicago's rank remained constant and U of Washington moved up one notch. All in all not very different than what I would have predicted and not unrealistic. I expect, as one moves down the list, the variance of the rank increases (probably to the point of being random at some point) and that should be reflected in the width of the interval provided by the NRC analysis.

Commercial magazines should not rank colleges

Anyone who has been to higher education knows that academia changes extremely slowly. So any ranking that alters the order of colleges dramatically differently from year to year is not worthy of studying by anyone, let alone the National Research Council. It makes for great drama (thus more profitable for your magazine) that Stanford, e.g., is #3 in one year but plummeted to #15 the following year, and Harvard is #1 this year but fell out of the top 10 altogether the next. But such drama is highly suspect among educators, and the academe (including the NRC) does operate on a different truth plane than that used by commercial businesspeople. You can devise a valid statistical machine for analysis, but if you don't take the time to examine your data, then it's just garbage in and garbage out.

response to TS of CA

The beauty of multiple regression is that it does not matter whether the rankings are randomly generated by a computer, by a three year-old playing with blocks or by the most thoughtful and error free methodology available. In a regression analysis the rankings would be included as another covariate in the regression model and we let the data inform us as to the predictability of the approach. The US News rankings (whether or not the methodology is understood) are available and well known to the public. I maintain that it would be an interesting empirical exercise (and an obvious one) to examine the association between the two approaches for each field. The

remark "According to the U.S. News Editor Brian Kelly, the magazine's college ranking formula is 'proprietary',so there is no way NRC or anyone can compare it" indicates a lack of understanding of what multiple regression does and how it works.

To do the empirical analysis you don't need the ranking formula-just the rankings.

Response to previous comment

To "statman" of XX:

If you have specific concerns about the NRC methodology, I hope you will contact that organization if you haven't already, so that changes/corrections can be made. Maybe they could add some footnotes, caveats, etc. before the final report is released.

Regarding your comment about NRC's supposed "arrogance" for ignoring the US News rankings, maybe it's not arrogance. Maybe it's because they (as well as academics and administrators) simply do not understand the formula that the magazine uses each year to come up with the rankings. The formula seems to change from year to year, resulting in a different order of the same colleges. These formulas are not transparent and explained as clearly as the NRC methodology, rendering them unusable to anyone. You might want to read the criticism of such practices here:

http://www.stanford.edu/dept/pres-provost/president/speeches/961206gcfallow.html

and here

http://www.washingtonpost.com/wp-dyn/content/article/2007/08/16/AR2007081602537.html

According to the U.S. News Editor Brian Kelly, the magazine's college ranking formula is "proprietary", so there is no way NRC or anyone can compare it or use it in any meaningful way. In order for NRC to look at those data, the data have to be available for viewing in the first place.

my 2 cents

I have an advanced degree in statistics and I found it a very tough read to figure out exactly what they did. I can't imagine how anyone without serious statistical training at the MS or PhD level can really understand what was done. I still haven't finished digesting the report but I do have some concerns. A very basic issue is the reliance on averages. In the most basic stats class we are taught that the average is not robust to outliers. So, for example, if the faculty in a department tend not to have many citations, with the exception of a "hot shot" the average will be unduly affected by this value. The median would have been better- this is expecially true for smaller departments. Even averaging the rankings can be problematic- think of a department that is not mainstream--- some reviewers love it and some hate it. The average (as well as the median) completely misses the boat. Beyond that, it would have been interesting if they had included department size as a variable in the regression analysis. Also, given the availability of US News graduate rankings, they could have also regressed on this variable to see if there is any association. (The rankings use different methods, but in the end what is the degree of the statistical association?) I would have also been interested in regressing on the US News undergraduate ranking. I found it somewhat arrogant on the part of the NRC to have ignored these data. (My personal bias is that there will be quite good correlation with the departments in the top 10-15 and then drop as the ranking of the weaker departments is pretty much random.) Finally, if you look on page 22, the "combined coefficient" for the variable "percent female students" is negative. On the surface that means that all other things equal, a department with more female grad students will be ranked lower than a department with fewer female graduate students. If they presented that example, there should have been some discussion in the report of that point. Was it an artifact of multicollinearity? Wait till the media picks up on that!

Respected rankings

The NRC rankings are the most widely respected rankings. The issue is not to provide a quick list but rather to carefully highlight specific programs. The changes from one year to another are usually not significant, aren't they Bob? After all, you should be very familiar with that aspect of the rankings.

Of course, since the NRC rankings do not have to deal with the discredited Peer Assessment score that you seem to favor, their validity trumps the US News rankings. Respected academicians and professionals know that and after all, that's what really matter.

I can't wait for the US News rankings 2010 of colleges. As usual, they should give me a good laugh. I hope that you do not disappoint us this year either, especially in light of all the scandals about scores manipulation: U of Florida, Clemson, USC plus all the ones that have not made the press ...yet.

Brief reply to "doctor"

First of all, anyone who bills themselves as "Doctor" has provided all the personal information I need about them.

Second, I think people who make judgments about the ratings you produce without really examining them have provided a lot of information about their research skills, too.

I can’t speak to the graduate ratings, but with regard to the undergraduate information the basic data ( SATs, acceptance rates, etc.) is simply a faster version of IPEDS. Same information, but you get it quicker. The company that does your reputational ratings can be proud of their work. It’s really pretty impressive. This is similar to the NRC ratings (old ratings, of course) the schools at the top are more consistently rated than the schools at the bottom. Everything considered, it’s topnotch.

You do a great job, Bob. Keep doing it.