Promoting Reliability and Replicability in Empirical Legal Research

Editors of law journals frequently solicit the expertise of Harvard Law School’s Empirical Research Department to conduct reviews of empirical articles. Traditionally, these reviews have focused on assessing the author’s selection of data and methodologies and evaluating the validity of the study’s design, assumptions and conclusions. This evaluation process typically operates under the presumption that the results reported by the authors are accurate and reliable. In recent years, however, there has been a noticeable rise in requests for replication or reproduction of study results as part of the review process. Distinct from a traditional review, a reproduction entails testing whether the results of a study can be replicated using the original data, while replication entails testing whether the results can be repeated using new data.  

In both the natural and social sciences, new research generally builds upon the foundation of prior empirical studies. Testing and verifying published research is thus fundamental to the progression of scientific knowledge, as it ensures the credibility of research findings. Rise in replicability and reproducibility within the legal scholarship reflects a growing commitment to transparency and rigor in the verification of empirical research. While these practices are relatively novel within the field of empirical legal studies, they are well-established in other academic disciplines. For example, in political science, most leading journals maintain replication archives for published articles, with 75% requiring authors to disclose data and 21% encouraging authors to provide data for replication (Brodeur et al., “Promoting Reproducibility and Replicability in Political Science,” 2024).  

The legal field, however, has been slow in adopting these practices. An examination of the top 30 faculty-edited legal journals in 2021 revealed that only two required authors to include a statement regarding the availability of their data, materials, and code. Only three journals required some form of transparency or an explanation from the author justifying non-compliance. The student-edited journals fared even worse, with 25 of the top 30 making no mention of any transparency guidelines (Jason Chin and Kathryn Zeiler, “Replicability in Empirical Legal Research,” 2021).  

Publication of bogus or flawed empirical papers contributes to a widespread replication crisis within the legal and other fields. A substantial number of replication or reproduction attempts fail to verify the claims of original studies. Some published assessment studies in the social sciences report reproducibility rates below 50%, with some findings even in the single digits. A 2022 review of the empirical legal literature exposed a lack of compliance with, and even little awareness of, the rules of inference, concluding that the “current state of empirical scholarship is deeply flawed” (Lee Epstein and Gary King, “The Rules of Inference,” 2022). Several other studies assessing the empirical legal scholarship observed similar trends (e.g. Abigail Matthews and Jason Rantanen, “Legal Research as a Collective Enterprise: An Examination of Data Availability in Empirical Legal Scholarship” (2022); Chin et al, “The transparency of quantitative empirical legal research published in highly ranked law journals (2018–2020): an observational study,” 2024). 

While law journals have been relatively slow in adopting best practices for data-driven research, recent years have seen encouraging developments. An increasing number of journals have started requiring authors to share study materials, with some prominent journals implementing comprehensive data transparency guidelines. These trends may encourage other journals to adopt similar practices. Enhancing transparency in research is a critical element of the ongoing credibility revolution occurring across various fields. Replications and reproductions of existing empirical studies are essential for the academic community to rigorously uphold the standards and best practices of statistical inference. To advance the legal discipline, replication and greater transparency should become integral components of the discourse within empirical legal studies.