A Change for the "Challenge Index" High School Rankings

In his article "Why I Changed the Challenge Index," Washington Post reporter and columnist Jay Mathews explains that if he hadn't modified the methodology for his annual rating of high schools, there would have been underperforming schools at the top of the 2008 Washington Post high school rankings.

The schools in question—those whose students take lots of AP and IB exams but pass very few of them—wouldn't make it on U.S.News & World Report's "America's Best High Schools" list either. Our methodology is based on the key principles that a great high school must serve all its students well (not just those who are bound for college) and that it must be able to produce measurable academic outcomes to show that it is successfully educating its student body. Mathews's change is a start, but it does not go far enough.

Mathews says that he has been ranking high schools for 10 years based solely "on participation in AP, International Baccalaureate, and other college-level exams. I call this the Challenge Index. It is the system used by Newsweek in its annual list of top high schools and by The Washington Postin its annual ratings of all Washington area schools."

The main criticism of the Challenge Index has been that it does not take into account whether students actually pass these college-level exams, just how many take them. Experts contend that the Challenge Index doesn't foster learning and academic achievement; it just encourages schools to offer a lot of these tests.

In his new methodology, Mathews places schools with test-passing rates below 10 percent on a "Catching Up" list. He goes on to say:

"Readers who have long advised me to remove low-scoring schools from the list altogether might interpret this change as acceptance of their point of view, and ask why I lack the courage to go all the way. This thinking is part of the new U.S.News & World Report "America's Best High Schools" list, inspired by Education Sector think tank co-founder Andrew Rotherham, with whom I have discussed the issue many times. Some might suggest that I am tilting in Rotherham's direction, but they would be wrong. He thinks that high schools like Coolidge (DC), Bell Multicultural (DC) and Crossland (MD) should never appear on any list of the nation's best schools, while I think they deserve public recognition, which the Catching Up list provides, for their strenuous efforts to change their cultures."

The bottom line is that Mathews's revised methodology still does not rate high schools on how well their students do on AP and IB tests, which is a key part of what the U.S. News rankings are based on. Until Mathews realizes that quality (passing tests) matters much more than quantity (how many are taken), his system will remain flawed and will encourage behavior by high schools that doesn't promote actual student learning.