Abstract

In “Cause, Effect, and the Structure of the Social World” (2023), Megan Stevenson makes a claim that randomized controlled trials (RCTs) have not had a significant effect in criminal justice settings. She then draws the conclusion that the gold standard for research designs, RCTs, are inherently incapable of doing so, demonstrating that the social world they intervene on is too complex, but also too resilient, to respond to the types of interventions that are evaluable by RCT. She calls the insistence that RCTs can work an “engineer’s” view of the world, which she discards as a myth. The argument then conflates RCTs with other methods of generating and sustaining change in organizations and systems, and closes suggesting RCTs should be discarded for less rigorous but more sweeping means of social reform. This article proceeds as follows: It characterizes Stevenson’s argument, which she asserts is empirical, as a de facto meta-analysis of criminal justice RCTs executed as a heuristic and presented in a narrative format. It argues that if a formal meta-analysis would be rendered invalid by violating established protocols, then a heuristic analysis that commits the same errors would be invalid as well. The analysis then presents the prohibitions on pooling studies with heterogeneous designs, interventions, outcomes, and metrics for the purpose of meta-analysis. It demonstrates that Stevenson pools a wide range of heterogenous studies, rendering her empirical meta-analytic claims problematic. It is true that many criminal justice RCTs have produced null or lackluster results—which also constitute an important outcome—and attempts to replicate significant findings have often been unsuccessful. This is not unique to criminal justice: psychology was recently in crisis when it was determined few of its most prominent studies could be replicated. However, less rigorous methods of reform do not solve this problem. Instead, more comprehensive research designs such as hybrid implementation/effectiveness trials can reveal aspects of our social world that impact external validity and generalizability. Findings from these studies can help illuminate the conditions that impact outcomes and sustainably modify highly resilient human behaviors. These methods arise from techniques in medicine and public health, which Stevenson brackets off as fundamentally different from criminal justice. This type of thinking may be the actual myth that prevents progress.