Selective labels: evaluating a prediction model on data from past decisions

In brief

Before a trial, a judge decides whether to release a defendant or to detain the defendant in jail. A released defendant can fail to appear in court, and we call this failure to appear (FTA). A model predicts the risk of FTA from the recorded features of a case, but the data show FTA only for released defendants. In court, judges can also notice things that the data do not record, which we call judge-only information.

• The release rate is the share of defendants released.
• The failure rate is the number of defendants who fail to appear, divided by all defendants, released or detained.
• A model rule releases a chosen share of the defendants: those with the lowest predicted risk. Its naive evaluation fills in each missing outcome with the predicted risk.
• The contraction estimate is the failure rate of the contraction rule. This rule lets the model detain more of the defendants whom the most lenient judge released. So the estimate uses only observed outcomes.
• Only the simulation knows the true failure rate of the model rule.

Move the sliders and watch these numbers. The explanation below the simulation gives the reasons.

The idea

Before a trial, a judge decides whether to release a defendant or to detain the defendant in jail until the trial. A released defendant can fail to appear in court. We call this failure to appear (FTA). Kleinberg et al. (2018) study these decisions with pretrial release data from New York City. Their main outcome is FTA. For convenience, the paper calls this outcome crime.

A prediction model uses the recorded features of a case, for example the current charge and the prior record, to predict the risk of FTA. To train and test the model, we need cases with a known outcome. We call the known outcome a label.

A detained defendant cannot fail to appear, so nobody sees what that defendant would have done after release. The data have labels only for the defendants whom the judges released. Lakkaraju et al. (2017) call this the selective labels problem. It is a form of selection bias: the decisions under study choose which outcomes we observe (Barocas et al. 2023).

A judge or a model decides whom to release. We describe each set of decisions with two numbers. The release rate is the share of defendants released. The failure rate is the number of defendants who fail to appear, divided by all defendants, released or detained (Lakkaraju et al. 2017). A detained defendant counts as no FTA. With this denominator, we can compare decisions with different release rates.

In symbols, write \(x\) for the recorded features and \(z\) for the judge-only information. Write \(D = 1\) if the judge released the defendant and \(D = 0\) if the judge detained the defendant. Write \(Y(1)\) for FTA if released (1 for FTA, 0 otherwise). A rule releases a set \(S\) of the \(N\) defendants. Its failure rate is:

\[\text{failure rate} = \frac{1}{N} \sum_{i \in S} Y_i(1)\]

A model rule releases a chosen share of the defendants: those with the lowest predicted risk. To compute its failure rate, we need the outcome of every defendant that it releases. The judges detained some of these defendants, so their outcomes are missing. A simple approach fills in each missing outcome with the predicted risk from the model. The model learned this prediction from released defendants with the same recorded features. We call this approach the naive evaluation.

In symbols, let \(m(x)\) be the predicted risk from a model fit on released defendants. The naive evaluation uses \(m(x)\) in place of \(Y(1)\) for the defendants whom the judges detained:

\[\begin{aligned} \text{naive evaluation} = \frac{1}{N} \Bigl[ & \sum_{i \in S,\, D_i = 1} Y_i(1) \\ & + \sum_{i \in S,\, D_i = 0} m(x_i) \Bigr] \end{aligned}\]

Judges see the defendant in court. They can notice things that the data do not record. The example in Kleinberg et al. (2018) is a gang tattoo. We call such information judge-only information.

Suppose that judge-only information predicts FTA, and that judges detain defendants partly because this information signals a high risk. Then the detained defendants have a higher risk than released defendants with the same recorded features. The filled-in outcomes are too low, and the naive evaluation makes the model look better than it is. More adjustment for the recorded features does not remove this error. The error comes from information that the data do not record.

So the naive evaluation rests on an assumption: among defendants with the same recorded features, released and detained defendants have the same average risk. This is the assumption of no unmeasured confounding, also called selection on observables. Judge-only information that predicts FTA and drives decisions breaks it (Kleinberg et al. 2018).

The problem is one-sided (Kleinberg et al. 2018). Suppose that a rule detains a defendant whom the judge released. We observed the outcome of that defendant, so we can compute the new failure rate. Suppose instead that a rule releases a defendant whom the judge detained. Then nobody observed the outcome.

Contraction uses the side that we can measure (Lakkaraju et al. 2017). The leniency of a judge is the release rate of that judge. Take the most lenient judge. Start from the defendants whom this judge released. The model detains more of them, highest predicted risk first, until the release rate falls to a target rate. Everyone whom the judge detained stays detained.

We call the result the contraction rule. The judge released every defendant whom the contraction rule releases, so we observe all of their outcomes. No outcome needs to be filled in. Then we compare the failure rate of the contraction rule with the failure rate of a stricter judge who has the same release rate. Kleinberg et al. (2018) apply the same idea to groups of judges sorted by leniency.

Contraction needs three things: many judges, cases assigned to judges as good as at random, and judges who differ in leniency. As good as random means that each judge hears a comparable set of cases. In real courts, assignment is as good as random only within cells. A cell is a group of cases, for example the cases heard in the same borough, year, month, and day of the week. Kleinberg et al. (2018) check this with balance tests: they test whether the predicted risk of defendants differs between more and less lenient judges. Contraction does not assume that judges ignore judge-only information.

Contraction also has limits. It can evaluate only release rates at or below the release rate of the most lenient judge. It never releases a defendant whom that judge detained. So it evaluates the contraction rule, not a model rule that ranks all defendants.

Accuracy on the labeled cases does not answer the question either. A common accuracy measure is the area under the receiver operating characteristic curve (AUC). Take a randomly chosen case with the outcome and a randomly chosen case without the outcome. The AUC is the probability that the first case has the higher predicted risk. A high AUC on the labeled cases does not show that the model makes better release decisions than the judges (Kleinberg et al. 2018).