Faculty Bibliography

The use of Artificial Intelligence (AI) based on data-driven algorithms has become ubiquitous in today's society. Yet, in many cases and especially when stakes are high, humans still make final decisions. The critical question, therefore, is whether AI helps humans make better decisions as compared to a human alone or AI an alone. We introduce a new methodological framework that can be used to answer experimentally this question with no additional assumptions. We measure a decision maker's ability to make correct decisions using standard classification metrics based on the baseline potential outcome. We consider a single-blinded experimental design, in which the provision of AI-generated recommendations is randomized across cases with a human making final decisions. Under this experimental design, we show how to compare the performance of three alternative decision-making systems--human-alone, human-with-AI, and AI-alone. We apply the proposed methodology to the data from our own randomized controlled trial of a pretrial risk assessment instrument. We find that AI recommendations do not improve the classification accuracy of a judge's decision to impose cash bail. Our analysis also shows that AI-alone decisions generally perform worse than human decisions with or without AI assistance. Finally, AI recommendations tend to impose cash bail on non-white arrestees more often than necessary when compared to white arrestees.

Despite an increasing reliance on fully-automated algorithmic decision-making in our day-to-day lives, humans still make consequential decisions. While the existing literature focuses on the bias and fairness of algorithmic recommendations, an overlooked question is whether they improve human decisions. We develop a general statistical methodology for experimentally evaluating the causal impacts of algorithmic recommendations on human decisions. We also examine whether algorithmic recommendations improve the fairness of human decisions and derive the optimal decision rules under various settings. We apply the proposed methodology to the first-ever randomized controlled trial that evaluates the pretrial Public Safety Assessment in the United States criminal justice system. Our analysis of the preliminary data shows that providing the PSA to the judge has little overall impact on the judge’s decisions and subsequent arrestee behaviour.

During the COVID-19 pandemic, online dispute resolution (ODR) systems became a popular method for courts to dispense justice while, allegedly, minimizing cost. Observing this growth, proponents have argued that ODR increases litigants’ access to justice, mitigates risks of procedural error, and conserves judicial resources. This Essay argues that each contention is empirically uncertain. ODR’s purported benefits lack empirical proof and likely depend on the platform’s design.This Essay recounts the Access to Justice Lab’s efforts to conduct two randomized control trials (RCTs) evaluating court-based ODR. Frustrated in our desire to contrast ODR to no ODR, or the availability of ODR to no such availability, we instead randomized (i) supplemental information about an ODR platform on citations versus no such information, and (ii) postcard encouragements to use an ODR platform versus no postcards. We were not surprised when one RCT saw only a single participant enroll over several months before we closed the study; nor were we surprised that, in the second RCT, the postcard failed to encourage ODR usage. However, we discovered that the presence of ODR boosted the efficacy of encouragements for users to resolve their traffic citations, whether users did so with the ODR platform or not. This boost comfortably surpassed the expected magnitude of similar encouragements in the literature, leading us to hypothesize the possibility of an interaction effect between ODR and reminders that may exceed the effect of the latter alone. We encourage additional research into this effect, and the broader impacts of ODR platforms.

Millions in the United States have criminal records. Many of these records are eligible for some sort of concealment from public view, commonly known as expungement or sealing. In this paper, we analyzed criminal records in four counties in Pennsylvania and several counties in Kansas to determine the number of records eligible for such remedies. In Pennsylvania, the analysis included both expungement, defined here as petition-based suppression of information, and sealing, defined here as suppression that the government (usually the judicial system) undertakes without petitions. Kansas law only allows for petition-based expungement. Our analysis found approximately 100,000 charges eligible for expungement in Kansas and 180,000 charges eligible for expungement in Pennsylvania, supporting prior research that identified a so-called “second chance gap.”Our primary contribution, however, is an analysis of which statutory reforms would provide the biggest bang for the buck, i.e., would render the largest number of cases or charges eligible for a record-clearing remedy. We found, for example, that elimination of criteria related to legally imposed financial obligations (“LIFOs”) would render a surprising number of files eligible for information suppression. In addition, our analysis identified approximately 200,000 charges that were eligible for sealing in Pennsylvania but were still available to the public online at the time of the data retrieval, suggesting that even when the government undertakes information suppression from its own databases, it finds the task challenging. Finally, our analysis examined why certain records were not eligible for expungement or sealing in each state. This additional analysis will inform legislatures and activists where their efforts can best be put to use.

Should the justice system sustain remote operations in a post-pandemic world? Commentators are skeptical, particularly regarding online jury trials. Some of this skepticism stems from empirical concerns. This paper explores two oft-expressed concerns for sustaining remote jury trials: first, that using video as a communication medium will dehumanize parties to a case, reducing the human connection from in-person interactions and making way for less humane decision-making; and second, that video trials will diminish the ability of jurors to detect witness deception or mistake. Our review of relevant literature suggests that both concerns are likely misplaced. Although there is reason to exercise caution and to include strong evaluation with any migration online, available research suggests that video will neither materially affect juror perceptions of parties nor alter the jurors’ (nearly nonexistent) ability to discern truthful from deceptive or mistaken testimony. On the first point, the most credible studies from the most analogous situations suggest video interactions cause little or no effect on human decisions. On the second point, a well-developed body of social science research shows a consensus that human detection accuracy is only slightly above chance levels, and that such accuracy is the same whether the interaction is in person or virtual.

Some have informally suggested that the process of obtaining an expungement order from the Kansas court system is simple and straightforward. We review the process for obtaining such an order in Kansas, and demonstrate the falsity of this suggestion.

Algorithmic recommendations and decisions have become ubiquitous in today's society. Many of these and other data-driven policies are based on known, deterministic rules to ensure their transparency and interpretability. This is especially true when such policies are used for public policy decision-making. For example, algorithmic pre-trial risk assessments, which serve as our motivating application, provide relatively simple, deterministic classification scores and recommendations to help judges make release decisions. Unfortunately, existing methods for policy learning are not applicable because they require existing policies to be stochastic rather than deterministic. We develop a robust optimization approach that partially identifies the expected utility of a policy, and then finds an optimal policy by minimizing the worst-case regret. The resulting policy is conservative but has a statistical safety guarantee, allowing the policy-maker to limit the probability of producing a worse outcome than the existing policy. We extend this approach to common and important settings where humans make decisions with the aid of algorithmic recommendations. Lastly, we apply the proposed methodology to a unique field experiment on pre-trial risk assessments. We derive new classification and recommendation rules that retain the transparency and interpretability of the existing risk assessment instrument while potentially leading to better overall outcomes at a lower cost.

We conducted a field experiment in which 311 low-income individuals seeking a divorce were randomly assigned to receive access to a pro bono lawyer (versus minimal help) to assist with filing for divorce. Examining court records, we found that assignment to an attorney made a large difference in whether participants filed for and obtained a divorce. Three years after randomization, 46% of the treated group had terminated their marriages in the proper legal venue, compared to 9% of the control group. Among “compliers”—participants who obtained representation only if assigned to receive it—those with lawyers were far more likely to file for and obtain a divorce than those not assigned lawyers. Because divorce implicates fundamental constitutional interests and can be effectuated only by resort to the courts, the US Constitution requires that dissolution of marriage be made achievable regardless of ability to pay. Yet, we observed few low-income individuals who were able to initiate divorce suits on their own. Through interviews and archival research, we identified barriers that low-income litigants faced in navigating the divorce system, including mandatory wait times, limited hours at important facilities, and burdensome paperwork sometimes requiring access to photocopiers and typewriters. This study therefore documents a salient instance in which a civil legal process was inaccessible to those without lawyers, even though their legal issues were straightforward, involving few if any matters for courts to adjudicate.

The importance of evidence-based policy rooted in experimental methods is increasingly recognized, from the Oregon Medicaid experiment to the efforts to address global poverty that were awarded a 2019 Nobel Prize. Over the past several decades, there have been attempts to extend this scientific approach to legal systems and practice. Yet, despite progress in empirical legal studies and experimental social policy research, judges, lawyers, and legal services providers often fail to subject their own practices to empirical study or to be guided by empirical data, with a particular aversion to randomized controlled trials (RCTs) (1). This is troubling, as many questions fundamental to legal practice and those it affects, such as allocation of attorney services, bail decisions, and use of mandatory mediation, could and should be informed by a rigorous evidentiary foundation. Although there are practical obstacles to undertaking legal practice RCTs, they have also been stymied by cultural barriers...

The United States legal profession routinely deals with evidence in and out of courtrooms, but the profession is not evidence-based in a scientific sense. Lawyers, judges, and court administrators make decisions determining the lives of individuals and families by relying on gut intuition and instinct, not on rigorous evidence. Achieving access to justice requires employing a new legal empiricism. It starts with sharply defined research questions that are truly empirical. Disinterested investigators deploy established techniques chosen to fit the nature of those research questions, following established rules of research ethics and research integrity. New legal empiricists will follow the evidence where it leads, even when that is to unpopular conclusions challenging conventional legal thinking and practice.

We report here the results of an RCT evaluating the effectiveness of a pro bono initiative’s oversubscribed divorce practice in Philadelphia County from January of 2011 until, effectively, July of 2016. The legal subject area was divorce, the quintessential example of a constitutional right that can be effectuated only by resort to the courts. Our study randomized an individual seeking assistance to pursue a divorce to either an effort by the service provider to find a pro bono attorney to represent her (treated group) or a referral to existing self-help or low bono resources coupled with an offer to answer questions by telephone (control group). Our study partner was the provider of last resort for free legal services in the Philadelphia County: it accepted intakes primarily via referrals from other organizations, and it required that service seekers exhaust all other options. Treated and control groups experienced different outcomes. If one limits one’s focus to Philadelphia County, where state venue laws “required” study participants and their opposing spouses to file, and where filing should have been most convenient for our study participants (who were all Philadelphia County residents), then we observe the following. Eighteen months after randomization, 54.1% of the treated group, as opposed to 13.9% of the control group, had a divorce case on record. Three years after randomization, 45.9% of treated group, as opposed to 8.9% of the control group, had achieved a termination of a marriage. The p-values for these differences (representing the probabilities that one would observe the numbers we observed, or numbers more extreme, if there were in fact no true difference between treated and control groups) were so low as to make them almost impossible to estimate; effectively, we observed instances of p = 0. If one expands one’s focus to other Pennsylvania counties, and thus considers filings by Philadelphia County residents who risked a dismissal due to improper venue and who abandoned the system they support as taxpayers, results remain statistically and substantively significant: 60.8% of the treated group, versus 36.3% of the control group, had a divorce case on file after 18 months, p < .00002; 50.0% of the treated group, versus 25.3% of the control group, succeeded in terminating the marriage in 36 months, p < .00002. When we account for the block randomization scheme we deployed, estimated effect sizes are a few percentage points larger than the numbers above would suggest. We conduct modeling to determine the effect of having a lawyer for divorce-seekers as a way of measuring the pro se accessibility of the divorce system. We find large effects, suggesting that the system is not accessible to pro se litigants.

We will never have enough lawyers to serve the civil legal needs of all low- and moderate-income (LMI) individuals who must navigate civil legal problems. A significant part of the access to justice toolkit must include self-help materials. That much is not new; indeed, access to justice commissions across the country have been actively developing pro se guides and forms for decades. But the community has hamstrung its creations in two major ways. First, by focusing these materials on educating LMI individuals about formal law, and second, by considering the task complete once the materials are available to self-represented individuals. In particular, modern self-help materials fail to address many psychological and cognitive barriers that prevent LMI individuals from successfully deploying their contents. This Article makes two contributions. First, we develop a theory of the obstacles LMI individuals face when attempting to deploy professional legal knowledge. Second, we apply learning from fields as varied as psychology, public health, education, artificial intelligence, and marketing to develop a framework for how courts, legal aid organizations, law school clinics, and others might re-conceptualize the design and delivery of civil legal materials for unrepresented individuals. We illustrate our framework with examples of reimagined civil legal materials.

After an appreciation of the contribution of the main text to the clarification and deepening of the utility and dilemmas of randomized clinical trials (RCTs) in medicine, this chapter notes the present ubiquity of RCTs in, for instance, social welfare programs, labor economics, education, political science, sociology, and law, several of which are discussed in detail. The chapter notes ways in which these are and are not like RCTs in the context of medical care. In several of the law examples, such as randomizing bail conditions and assigning lawyers to meet legal needs of low-income individuals, none of the subjects of the research are (yet) in a client relationship similar to that of a patient, and so there is no analogous duty on the part of the experimenters. It is an allocation of scarce resources in part carried out in a way that may yield more reliable knowledge.

This essay discusses the importance of outreach and intake in the role that legal services providers fill in the current U.S. legal system, as well as how little is known about either subject.

Routine default threatens the foundations of the United States court system. We study the problem of routine default by human defendants, using the Boston Municipal Court’s debt collection docket as our laboratory. Arbitraging various non-law literatures, we designed interventions consisting of two forms of mailings. We study the effectiveness of our two mailings in a randomized control trial that includes a no-intervention control group. We find no difference in effectiveness as between our two mailings, but that both roughly double the rate at which defendants participate in their lawsuits. Results are statistically significant. We discuss implications of our findings.

Research on low-cost ways to engage consumers holds promise for tackling the high default rates in debt-collection lawsuits.

This article describes a proposed randomized control trial (RCT) involving individuals in financial distress, specifically, individuals sued on a credit card debt collection case by a debt buyer or creditor. The aim of the RCT is to evaluate the effectiveness of two interventions often proposed to help individuals in financial distress improve their financial health. We intend to test, inter alia, whether (1) an offer of legal representation in the debt collection case, (2) an incentive to undergo the same financial counseling required in bankruptcy, or (3) a combination of both treatments have an effect on the financial health of financially distressed consumers. Our primary outcome measure will be credit scores and reports, although we also aim to survey study participants about other outcomes such as changes in health, assets, and general well-being. We describe in detail not just the methodology of the study, but also the mechanics of how we have gone about executing this complex field experiment so far. The objective in doing so is to demystify the process for scholars, legal aid lawyers, and clinical professors who have not had direct experience with empirical methods, and to encourage them to think about conducting RCT evaluations of their own programs. The article also describes three other goals we seek to advance through the project. The first is a deeper understanding of debt collection in the courts. Allegations of abuse abound about the industry generally and legal collections specifically, but to date, the evidence is largely anecdotal. We will be uniquely positioned to learn about whether the allegations are more than anecdotes. The second is increasing access to justice for unrepresented defendants in collection cases. Along with a team of dedicated law students, the study team is developing an attorney litigation manual and pro se materials, including court forms and scripts, for debt collection litigation. Before making the pro se materials available, they will be field tested with individuals in similar situations to those facing study participants. The third goal is developing an innovative pedagogical approach to legal instruction that combines doctrine and practice. A majority of the law students participating in the project are doing so in the context of a seminar which incorporates both doctrinal instruction about the consumer credit system and clinical experience. We hope that this article provides our readers both with ideas for future research projects and with ideas about how to incorporate diverse goals in a single study design.

We discuss the history of the exit poll as well as its future in an era characterized by increasingly effective and inexpensive alternatives for obtaining information. With respect to the exit poll's future, we identify and assess four purposes it might serve. We conclude that the exit poll's most important function in the future should, and probably will, be to provide information about the administration of the franchise and about the voter's experience in casting a ballot. The nature of this purpose suggests that it may make sense for academic institutions to replace media outlets as the primary implementers of exit polls.

We discuss the concept of "unbundled" representation in litigation matters, in which a client retains a lawyer to provide legal services short of those that would be provided in a traditional attorney client relationship. Specifically, we trace the evolution of the process of mainstreaming unbundled representation in three case study states: Colorado, Massachusetts, and Alabama. We conclude by providing a comprehensive bibliography of academic and some other writings on the subject of unbundled representation.

Despite their ubiquity, observational studies to infer the causal effect of a so-called immutable characteristic, such as race or sex, have struggled for coherence, given the unavailability of a manipulation analogous to a “treatment” in a randomized experiment and the danger of posttreatment bias. We demonstrate that a shift in focus from actual traits to perceptions of them can address both of these issues while facilitating articulation of other critical concepts, particularly the timing of treatment assignment. We illustrate concepts by discussing the designs of various studies of the role of race in trial court death penalty decisions.

Racial bloc voting is the central concept in judicial regulation of redistricting. For the past several decades, the definition and proof of this concept have depended on two premises: that polities can be conceptualized in biracial terms and that nearly perfect information on voting patterns can be inexpensively obtained from simple statistical methods. In fact, however, neither premise has been true for some time, as the nation has become multiracial and allegations have increased that Caucasians vote less monolithically than before, with both assertions imposing severe stress on the simple statistical methods previously used to assess voting patterns. In this article, I analyze these challenges to traditional understandings and attempt to answer the following question: how can we litigate racial bloc voting well in the current era? I provide recommendations, including greater reliance on more sophisticated statistical methods, an increase in the use of sample surveys, and a renewed receptivity to nonquantitative evidence on voting patterns, while clarifying that each of these recommendations carries substantial costs. I then discuss the conceptual and normative implications of my recommendations on the empirics.

The Civil Rights Movement had a variety of transformative effects on the way federal courts hear and decide cases; among them was the introduction of quantitative analysis as a staple of certain types of high-profile adjudication, particularly in redistricting cases. The first judicial foray into regulating the drawing of electoral districts-the "one person, one vote" line of cases-was premised on an equality norm expressed in explicitly numerical terms. In these cases, the Supreme Court settled on numerical guidelines requiring only simple arithmetic to implement. Since then, however, the federal judiciary has engaged with increasingly complicated quantitative measurements and statistical techniques, first in the racial vote dilution cases, then in the "overuse of race" cases, then in the partisan gerrymandering cases.

Despite its shortcomings, cross-level or ecological inference remains a necessary part of some areas of quantitative inference, including in United States voting rights litigation. Ecological inference suffers from a lack of identification that, most agree, is best addressed by incorporating individual-level data into the model. In this paper we test the limits of such an incorporation by attempting it in the context of drawing inferences about racial voting patterns using a combination of an exit poll and precinct-level ecological data; accurate information about racial voting patterns is needed to assess triggers in voting rights laws that can determine the composition of United States legislative bodies. Specifically, we extend and study a hybrid model that addresses two-way tables of arbitrary dimension. We apply the hybrid model to an exit poll we administered in the City of Boston in 2008. Using the resulting data as well as simulation, we compare the performance of a pure ecological estimator, pure survey estimators using various sampling schemes and our hybrid. We conclude that the hybrid estimator offers substantial benefits by enabling substantive inferences about voting patterns not practicably available without its use.

In Statistics Is a Plural Word, a response to my article Causal Inference in Civil Rights Litigation, Dean Steven Willborn and Professor Ramona Paetzold take issue both with my critique of regression as it is currently used in civil rights litigation and with my advocacy of the potential outcomes framework. In this Reply, I argue that Dean Willborn and Professor Paetzold’s response does not address (and thus cannot refute) the central lessons of Causal Inference, despite purporting to agree with those lessons. In particular, after “agree[ing] wholeheartedly” that a definition of a causal effect is necessary for the use of statistics in civil rights, Plural does not offer a definition. In the absence of such a definition, the purpose of statistics in civil rights litigation is unclear. The potential outcomes framework, in contrast, provides the needed definition and clarifies many subsidiary concepts, with salutary consequences following naturally from a start in the right place.

Despite its potential pitfalls, ecological inference is an unavoidable part of some quantitative settings, including US voting rights litigation. In such applications, the analyst will typically encounter two-way tables with more than two rows and columns. Although several ecological inference methods are currently available for 2×2 tables, there are fewer options for analysing general R×C tables, and virtually none that model counts as opposed to fractions. We propose a count R×C method that respects the bounds deterministically, that allows for complex relationships between internal cell quantities, that is easily extensible and that results from transparent assumptions. We study the method via simulation, and then apply it to an example that is drawn from the state of Texas relevant to recent redistricting litigation there.

Civil rights litigation often concerns the causal effect of some characteristic on decisions made by a governmental or socioeconomic actor. An analyst may be interested, for example, in the effect of victim race on jury imposition of the death penalty, in the effect of applicant gender on a firm’s hiring decisions, or in the effect of candidate ethnicity on election results. For the past thirty years, such analyses have primarily been accomplished via a statistical technique known as regression. But as it has been used in civil rights litigation, regression suffers from several shortcomings: it facilitates biased, result-oriented thinking by expert witnesses; it encourages judges and litigators to believe that all questions are equally answerable; and it gives the wrong answer in situations in which such might be avoided. These difficulties, and several others, all stem from the fact that regression does not begin with a paradigm for defining causal effects and for drawing causal inferences. This Article argues for a wholesale change in thinking in this area, from a focus on regression coefficients to an explicit framework of causation called “potential outcomes.” The potential outcomes paradigm of causal inference, which (for lawyers) may be analogized to but-for causation with a renewed emphasis on time, addresses many of the shortcomings of regression as the latter is currently used in civil rights litigation, and it does so within a framework courts, litigators, and juries can understand. This Article explains regression and the potential outcomes paradigm and discusses the latter’s application in the death penalty, employment discrimination, and redistricting settings.

Recent developments in law and in quantitative methods have combined to place greater emphasis on coherent and accurate techniques of drawing inferences about racial voting patterns in Voting Rights Act litigation. In this article, I examine the challenge the secret ballot poses for such inferences; I then discuss four so-called “ecological inference” methods designed to address the issue. I argue that the two techniques that have dominated this field for more than 20 years should be abandoned; that a third, well-publicized method should be used only when no other is feasible; and that a fourth, while representing the state of the art at present, has shortcomings that researchers should address. To aid in understanding, I apply each technique outlined to a concrete data set. I conclude with a discussion of what makes a good method.