Principal strata: when the treatment decides whose outcome we see

In brief

A city runs a job training program, and a coin flip decides who gets the training. The training can help some people find a job. A wage exists only for a person with a job, so we can compare wages only among people who have a job. We want to know how the training changes wages.

• The naive comparison is the average wage of trained workers minus the average wage of untrained workers.
• The always employed would work with or without the training. The true effect for the always employed is the effect of the training on their wages. Only they have a wage both with and without the training.
• The fair comparison compares the wages of the always employed only. Real data cannot make this comparison, because nobody can see which group a person belongs to.
• The Lee bounds are a lower and an upper bound for the effect on the always employed. They assume that the training never costs anyone a job.

Move the sliders and watch these numbers. The explanation below the simulation gives the reasons.

The idea

A city runs a job training program. A coin flip decides who gets the training. We want to know how the training changes wages. Because of the coin flip, the trained group and the untrained group are alike on average (see random assignment).

A wage exists only for a person with a job. So we can compare wages only among people who have a job. This looks harmless, because the coin flip assigned the training. It is not harmless, because the training can also change who has a job.

Suppose that the training helps some people find a job. Without the training, these people would not work. They may also earn less than people who would work anyway. Then the trained workers include extra workers with low wages.

A comparison of wages among workers can then make the training look as if it lowers wages. This can happen even when the training raises the wage of every worker. The problem comes from a comparison that conditions on a variable, employment, that the treatment itself changes. This is called post-treatment bias (Frangakis and Rubin 2002).

Four kinds of people

Each person has two potential employment outcomes: S(1), whether the person works with the training, and S(0), whether the person works without it. The pair of values sorts people into four groups. Frangakis and Rubin (2002) call these groups principal strata.

The training cannot change the group of a person, because each group is defined by both potential outcomes. So a comparison inside one group compares like with like. A wage effect is well defined only for the always employed. Only they have a wage both with and without the training (Zhang and Rubin 2003).

In symbols, write \(D = 1\) for a trained person and \(D = 0\) otherwise. \(S(d)\) is employment and \(Y(d)\) is the wage under training status \(d\). \(Y(d)\) exists only when \(S(d) = 1\). The target is the wage effect for the always employed (AE), the people with \(S(0) = S(1) = 1\):

\[\tau_{\text{AE}} = E[Y(1) - Y(0) \mid S(0) = 1, S(1) = 1]\]

Nobody can see which group a person belongs to, because each person shows only one of the two employment outcomes. A trained person with a job is always employed or helped. An untrained person with a job is always employed or hurt. So each observed group of workers is a mix of two strata. A comparison of their average wages then mixes the effect of the training with the difference between the strata.

In symbols, the naive comparison is \(E[Y \mid D = 1, S = 1] - E[Y \mid D = 0, S = 1]\). Suppose that the training never costs anyone a job. Under random assignment, the untrained workers are then all always employed, and the trained workers are the always employed plus the helped (H). Write \(\pi_{\text{AE}}\) and \(\pi_{\text{H}}\) for the shares of the two strata. Then

\[\begin{aligned} & \text{naive} - \tau_{\text{AE}} \\ = {} & \frac{\pi_{\text{H}}}{\pi_{\text{AE}} + \pi_{\text{H}}} \, (E[Y(1) \mid \text{H}] - E[Y(1) \mid \text{AE}]) \end{aligned}\]

The bias is zero when nobody is helped or when the helped earn, on average, what the always employed earn with training.

What the data can still tell us

Assume that the training never costs anyone a job. This assumption is called monotonicity. Then the hurt group is empty, and every untrained worker is always employed. The experiment also shows how many people the training helps: the employment rate with training minus the employment rate without it.

Lee (2009) uses these two facts to bound the wage effect for the always employed. Among trained workers, a known share is helped, but we do not know which workers they are. If the helped workers have the highest wages, dropping that share from the top of the trained wages gives a lower bound. If they have the lowest wages, dropping it from the bottom gives an upper bound. The true effect for the always employed lies between the two bounds. The bounds get wider when the training helps more people.

For the bounds, let \(q_T\) and \(q_C\) be the employment rates with and without training. The share of helped people among trained workers is \((q_T - q_C) / q_T\) (Lee 2009).