The U.S. News and World Report Decided to Reject a Proposed Law School Rankings Metric that Would Have Impacted LSA Scholars
AMHERST, Mass. – The Law and Society Association is pleased to report that the U.S. News and World Report has decided not to proceed with its proposal to add a measure of “scholarly impact” to its law school rankings. This report summarizes the Association’s concerns and actions taken while the metric was under consideration.
The proposed metric for measuring scholarly impact was to be drawn from citations in the Hein database which is comprised primarily of law review articles. The Law & Society Association was concerned with the implications for our members, many of whom write books, book chapters and peer-reviewed articles that appear in and are frequently cited in publications other than law reviews. For that reason, the scholarly impact of our members’ scholarship would not have been accurately represented in the proposed metric. In addition, non-doctrinal and data- driven sociolegal scholarship ordinarily does not amass as many citations as constitutional or purely doctrinal articles do. Moreover, if law schools attempted to game the system to improve their rankings, as has happened with other U.S. News metrics, the intellectual, interdisciplinary, and racial diversity of law school faculties and the range of topics they choose to write on might have been negatively affected.
The twin concerns of the scientific integrity of the proposed rating system and the negative impacts on some LSA members, led Law and Society Association past-President, Professor Kim Scheppele to call attention to the effort when the now-rejected metric was first proposed. LSA President, Dean Penny Andrews appointed a committee consisting of Richard Lempert, Elizabeth Mertz, and Rachel Moran to investigate further and to take action on the part of the Association. Although our sister organization, the Society for Empirical Legal Studies, issued a critique suggesting that USNWR use an alternative data base, the US News nevertheless proceeded to explore partnering with Hein to construct the new metric. So the LSA committee, which had been formed before the SELS letter, continued to investigate the proposed metric’s scientific merit and likely impacts and to consider ways of making its concerns known.
The committee communicated its initial concerns to key actors at U.S. News and World Report and at Hein in January 2020. It proposed and hosted a Hot Topics Program on “Ranking Legal Scholarship: U.S. News & World Report’s New Metric and Its Effect on the Legal Academy” at the January 2020 Annual Meeting of the Association of American Law Schools (AALS) in Washington, D.C. We welcomed the participation of Robert Morse, who headed the U.S. News effort, at that session.
In addition to Morse, representatives of Hein were present at this well-attended panel, and immediately following the session they spoke with LSA Presidents Scheppele and Andrews. It was clear that LSA’s concerns were shared by many legal academics. Intensive discussion with Morse and his team continued afterwards in a small group setting where we could dig into various issues and discuss the scientific challenges associated with the effort, especially concerns with the accuracy and potential biases of the Hein database, as well as likely impacts within law schools and on the Association’s members. Law professors and a law librarian who were involved in developing alternative metrics, as well as a representative from AccessLex, attended along with the LSA committee members.
We appreciated the conversations that occurred then and in some follow-up communications, including Hein’s willingness to work to improve their database by correcting erroneous author attributions and making it more inclusive of interdisciplinary scholarship. In particular, responding to widespread concerns expressed by law schools and librarians as well as our committee, Hein made a welcome plan to incorporate professors’ ORCID listings. The LSA Committee accordingly hosted a panel at our annual meetings in which a representative from ORCID along with a well-respected law librarian provided guidance on how to use ORCID. We here repeat the advice given at that panel: LSA members are advised to ensure that they have ORCID numbers and to be sure that this ID is associated with all their publications — for regardless of the decision that U.S. News recently made, securing credit for one’s writing via electronic searches can only increase in importance.
LSA was not alone in our effort to ensure the unbiasedness and soundness of the U.S. News rankings or to see the project cancelled if obstacles were too big to overcome. The Association is grateful for the extensive work of U.S. law librarians who were deeply engaged with U.S. News and Hein about the difficulties they discovered with the various proposed approaches to a legal scholarship ranking. The librarians helped educate us as a committee on the issues involved, and their willingness to keep LSA informed of developments in their interactions was crucial in our ongoing participation in relevant discussions, in planning panels, first at AALS and later at LSA, and in outreach to LSA members with relevant expertise (with help from LSA’s CRN on New Legal Realism). Scholars who identify primarily with the Society for Empirical Legal Studies (SELS) also contributed to the outcome. LSA is also grateful for help from ORCID, to which we turned upon learning about the planned collaboration with Hein, as we attempted to update our own members’ listings (many of whom are law professors performing empirical research).
Finally, the Association thanks both Robert Morse at U.S. News and the Hein organization who were willing to listen to us and others. Despite the intellectual investment they had made in their plans, they were willing to drop them when they could not overcome their shortcomings. In the click-happy world the media now live in, this is all too rare.
###