Can the methods of science help inform the practice of law? And can a tradition-bound profession be transformed into a field that uses evidence to determine which legal interventions are safe and effective, both for individuals caught up in the justice system and society as a whole?
For Harvard Law School Professor Jim Greiner at the Access to Justice Lab, the answer is yes. Judges and lawyers, he argues, must begin to rely more on evidence garnered from randomized clinical trials, rather than anecdote and personal experience, to ensure that the decisions they make have the results they intend. In doing so, he says, the legal profession should follow the path charted by medicine, which fitfully transformed itself from an art into a science over the course of the 20th century.
Harvard Law Today recently spoke with Greiner about the Access to Justice Lab’s work, and about an ongoing clinical trial it is overseeing in Dane County, Wisconsin to examine the value of risk assessment tools designed to inform judicial decisions about whether or not to release defendants awaiting trial. Greiner is the Honorable S. William Green Professor of Public Law at Harvard Law School. In addition to earning his J.D. from the University of Michigan Law School, he also completed a Ph.D. in statistics at Harvard University in 2007, the same year he joined the HLS faculty.
Harvard Law Today: What is the Access to Justice Lab and what does it do?
Jim Greiner: The Access to Justice Lab is a research center within Harvard Law School dedicated to transforming the legal profession in the United States, and maybe abroad, into an evidence-based field. Today, what counts as true in the legal profession is based on the personal experiences of individual lawyers and judges. But what doesn’t currently have a place at the table are the types of methods that scientists and statisticians have created to determine what is true and what does and does not work.
This includes randomized clinical trials, which are used in medicine to decide whether a new drug or a new medical device actually works, such as the trials currently underway to test COVID-19 vaccines. And we insist on these randomized trials to ensure the vaccines will work and won’t have devastating side effects. Medicine has made this transition from a personal experience-based field into an evidence-based field. Law has not. We think that’s bad and, as a result, that law is continuing to insist on a lot of things that likely either don’t work, or are counterproductive.
So, we are trying to transform law into an evidence-based field by conducting randomized field experiments, to create knowledge, to demonstrate how knowledge should be created and to challenge the prevailing hegemony within the legal profession. They require professionals like judges to temporarily give up control over the decision-making process and turn it over to a randomizer to decide, say, whether this individual needs this pretrial program versus that pretrial program. Getting the judge to say, ‘I don’t know which one of these two pretrial programs is better, so I’m going to turn the decision over to somebody else to figure that out’ is quite a gut check for a judge. So, getting judges into the mindset of using randomized trials to learn which intervention will produce better outcomes would go a long way towards transforming the legal profession into an evidence-based field.
HLT: Can you tell us a little bit about the randomized trial that you have going on in Wisconsin?
Greiner: Dane County, Wisconsin is adopting a series of measures to reform its pre-disposition criminal justice process, which is the time period between when an individual is arrested and when that person is adjudicated and, if found guilty, sentenced. As part of those reforms, they are adopting a risk assessment instrument to help judges decide whether to release the individual while they are awaiting trial or the disposition of their case (such as a plea agreement).
A risk assessment instrument is basically an actuarial analysis—an effort to use observable facts, such as past criminal history or age, to see how risky it will be if you allow this person to go free. Will they commit a new crime or fail to appear at a subsequent court proceeding? Or if they commit a new crime, will be a violent crime? The idea is to classify people according to the risk of misbehavior based on past statistics of similarly-situated people and then to provide that information to a judge or a decision-maker to help them make better decisions.
So, Dane County wanted to adopt a risk assessment instrument tool called the Public Safety Assessment–Decision Making Framework, or PSA for short, and asked us to help determine whether it will actually improve their decision making. The PSA was developed with support from Arnold Ventures, a Houston-based philanthropy, to reduce the burden placed on vulnerable populations at the front end of the criminal justice system. Our research is designed to determine how the PSA affects the adjudicatory decisions that affect those individuals. ‘Will it allow us to minimize or reduce the number of days we have to incarcerate people, the number of failures to appear, the number and severity of new crimes, especially new violent crimes, and differences between racial groups in our criminal justice administration? Can we simultaneously minimize all of those? Can we minimize some of them? If we’re able to minimize some, but we have to pay for it with increases in others, what does that look like? In other words, what is the risk assessment score doing for us in our community?’
The right way to figure that out is with a randomized control trial. So, as cases came before the court, we randomized whether the judge would or would not have the risk assessment instrument present in the decision making. And then we followed everybody to see what happened along these various dimensions: failures to appear, new criminal new crimes, new violent crimes, days people sent spent incarcerated while awaiting disposition of their cases. And by comparing the group for whom the judge was provided the risk assessment report to the group without it, we can tell what the effect of the risk assessment report is.
HLT: You recently issued an interim report. What have you found so far?
Greiner: The findings could change between now and when we issue a final report because we only have about 20% of the total data that we hope to have. But so far, we’re seeing no difference between the treated and the control group. One of the hotly contested issues in criminal law these days is whether risk assessment instruments embed and make permanent racial biases. So far, we’re not seeing very much difference on that or any other dimension. In other words, a lot of the rhetoric around these risk assessment instruments may not be justified, but we need the final results from this and other trials to see.
HLT: Are you doing any other studies on this topic?
Greiner: We are conducting other studies of this same risk assessment instrument, which will enable us to see if we can replicate the study more than once, which is another important scientific concept. Physicians, for example, repeat studies to make sure they didn’t just get an accidental result in one. We are doing the same thing. But so far, we’re so we’re showing no difference on measures of racial bias or the other dimensions I mentioned.
“What often passes for argument in law is: ‘My experience says that I’m right, let me tell you about my experience‘”
We’re also conducting a set of studies that are not RCTs on part of this risk assessment system. These are called “validation studies,” and they involve looking at past data to assess how well the risk assessment classifies individuals who are arrested and then released on risk of misbehavior. So, basically, we look at the people that the system said had lower risk and ask whether, for those who achieved release, did they misbehave at lower rates than, say, people whom the system said had higher risk. The statistical techniques we use are more sophisticated than that, but that’s the gist of it.
HLT: Are risk assessment instruments used in other areas, and if so, what are the results?
Greiner: Risk assessment instruments can be used when a cop decides who to stop on a street. Prosecutors could use them to help guide charging decisions. In criminal justice, they can be used for initial release and bail determination decisions after somebody is arrested, or in sentencing or parole. And by the way, it’s not like these are only used in criminal justice processes. They aren’t. They are frequently used in medicine to help doctors diagnose. They’re frequently used in the human resources setting to help decide who to hire. And they’re used in engineering decisions, to assess environmental risks, and in deciding which inventions people should invest in. In loan management, a credit score is basically a risk assessment instrument. So, there’s a long tradition of using these things.
HLT: There is a quote on the Access to Justice Lab website that reads: “In no field is resistance to empirical data stronger than in the in U.S. legal practice.” Why is that?
Greiner: I don’t know why it is that the United States legal profession has been so reluctant to accumulate and rely on scientific reasoning and scientific knowledge. Some have argued that, well, law is not a science. My problem with that was that in 1935, medicine wasn’t a science. Physicians consciously transformed their profession into a science and decided to make it rely more on scientific evidence. If you had uttered the phrase ‘medical science’ in 1935, nobody would have had a clue what you were talking about. And so, it was not an easy or gentle process to transition. And it was not a process that many physicians at the time welcomed. And in fact, many physicians currently resist the incorporation of what evidence tells them into their practical recommendations. So, saying that law is not a science is a result, not an explanation.
Other people have said it’s because lawyers are bad at math. Again, that argument seems circular to me. In 1935, you didn’t need a scientific background for medicine. But now, we insist on it as a precursor to getting into getting into medical school. It just shows the power of habit and social construction to say that randomized experiments and, indeed, science can’t really help us create programs that deal with human beings, whether it be risk assessment instruments or pretrial diversion programs. And if we’re not considering those sorts of scientific things, what are we using to make these decisions? Human instinct? That’s kind of terrifying.
My hope is that, because I can’t think of anything inherently structural about law that is different from medicine, we can change law into an evidence-based field, and maybe avoid doing some of the horrible things that we do to people in law. We have capital punishment. We incarcerate people for vast lengths of time, ruining their lives and the lives of their families around them. We police people, stick people in handcuffs and hit them. On the civil side, we make it impossible for people to get divorces when they don’t want to be married anymore. We deny them benefits they need to keep eating, to keep living.
There may be good justifications for doing all these horrible things. We may have to do them to avoid other costs that we’d rather not suffer. But let’s at least know. Let’s find out whether we need to be doing all these horrible things.
HLT: What drew you to looking at these types of empirical questions?
Greiner: So, it’s a little bit of a journey. I had no math or statistics courses in high school, college or law school. I did do pre-med but no math or statistics or anything like that. When I moved from the Justice Department to my private law firm for three years, I took on redistricting cases, and those involved a fair amount of heavy numbers. And I kept getting stuff that I didn’t believe from expert witnesses and from clients. I remember talking to an expert witness who was going to swear under oath that the best estimate from his statistical model was that 125% of the African American population voted for the Democrat in a particular election. And I said, ‘I don’t think that’s right. What’s the uncertainty interval around that?’ I was taking some math and statistics courses at the time. He said, ‘Plus or minus 3%’, then started to laugh because he knew that was garbage. So, I started taking more math and statistics courses, got bit by the bug, went back and got a Ph.D. in statistics, and then started researching.
What I found most interesting were these field experiments—and not just the experiments, but the incredible difficulty I was having persuading lawyers that evidence-based thinking was a thing, that it was useful. The overwhelming reaction was: ‘Why in the world would we study that? I already know the answer. It’s just that the idiot sitting next to me doesn’t know that I’m right. So, you need to focus on persuading the idiot sitting next to me that I’m right.’ My reaction was: “How am I going to do that? I don’t have any evidence. How do I know that you’re right and not the idiot sitting next to you? How do I know that you’re not the idiot?’
What often passes for argument in law is: ‘My experience says that I’m right, let me tell you about my experience.’ But of course, we know from medicine, from economics, from international development, and from dozens of interventions that experts have posited to be effective have turned out to make either no difference or to be very counterproductive. For instance, the idea of sending juveniles into prisons for a day or two to scare them straight turns out to be a great way to increase recidivism. Another example is microlending, the Nobel prize winning idea of giving small loans to would-be entrepreneurs in developing countries as a way to increase their household income and consumption. It doesn’t do either of those things. It may have other effects, but the original basis upon which it was sold doesn’t work.
I found that reliance on personal experience and lack of openness to the scientific process more fascinating than anything else. And I also wanted to apply that kind of thinking in an area where I wasn’t dealing with helping rich people save money. I’m more interested in how the law interacts with individual human beings, especially individual human beings who cannot afford to hire a lawyer to interact with the law for them.
HLT: Questions related to social, criminal and racial justice reform have gripped the nation in recent months. What can your research do to help inform those debates?
Greiner: The pendulum has swung away from tough on crime rhetoric and towards a discussion of mass incarceration. Many major jurisdictions are electing prosecutors running on platforms of incarcerating fewer people. That was unheard of 20 years ago. But I think the pendulum could swing back. And it could continue swinging back and forth unless we have evidence on how to do decarceration well. I think that’s an example of a debate we hope to inform. But we need to recognize that there needs to be a long-term investment, because good knowledge just doesn’t come easy.