Difference-in-differences: what breaks when parallel trends fails

In brief

A chain of stores starts a new policy in some of its stores in period 6. These stores are the treated group, and the other stores are the comparison group. We observe an average outcome, for example weekly sales, in both groups in periods 1 to 10.

• The difference-in-differences (DiD) estimate is the change in the treated group minus the change in the comparison group. Each change is the average in periods 6 to 10 minus the average in periods 1 to 5.
• The average treatment effect on the treated (ATT) is the average effect of the policy on the treated stores after the start. The simulation knows the ATT, so the readouts show it as the true effect. Under the assumptions in the explanation, the DiD estimate estimates the ATT.
• Estimate minus true effect shows how far the DiD estimate misses the ATT in the current sample.
• The estimated extra growth per period uses periods 1 to 5, before the start. It estimates how much faster the treated group grows than the comparison group in each period.

Move the sliders and watch these numbers. The explanation below the simulation gives the reasons.

The idea

A chain of stores starts a new policy in some of its stores in period 6. The other stores do not start the policy. We call the first set of stores the treated group and the second set the comparison group. We observe an average outcome, for example weekly sales, in both groups in periods 1 to 10.

To measure the effect, we need a number that nobody observes: the average outcome of the treated group after the start, without the policy. There are three common ways to fill in this number. Each way needs its own assumption (Cunningham 2021, ch. 9).

1. Use the comparison group after the start. This assumes that, without the policy, both groups would have the same average outcome.
2. Use the treated group before the start. This assumes that, without the policy, the outcome of the treated group would not change over time. A before-and-after comparison mixes the effect of the policy with everything else that changed over the same time. For example, the season or prices in the whole market may have changed.
3. Use the treated group before the start, plus the change in the comparison group. This is difference-in-differences.

Difference-in-differences (DiD) uses the comparison group to remove the second thing (Angrist and Pischke 2009, ch. 5; Cunningham 2021, ch. 9; Huntington-Klein 2025, ch. 18). It subtracts the change in the comparison group from the change in the treated group. Some textbooks write DD or DID for the same method.

\[\begin{aligned} \text{DiD} = {} & (\text{treated after} - \text{treated before}) \\ & - (\text{comparison after} - \text{comparison before}) \end{aligned}\]

The change in the comparison group stands in for the change that the treated group would have had without the policy. This step is correct only under the parallel trends assumption. This assumption is about the treated group without the policy. Its average outcome would have changed by the same amount as the average outcome of the comparison group.

In symbols, write \(Y(1)\) for the outcome with the policy and \(Y(0)\) for the outcome without the policy. \(E[\,\cdot\,]\) is an expected value, that is, an average over the population of stores. A vertical bar means among: \(E[Y(0) \mid \text{treated}]\) is the average of \(Y(0)\) among the treated stores. Parallel trends says:

\[\begin{aligned} & E[Y_{\text{after}}(0) - Y_{\text{before}}(0) \mid \text{treated}] \\ = {} & E[Y_{\text{after}}(0) - Y_{\text{before}}(0) \mid \text{comparison}] \end{aligned}\]

The two groups do not need the same level of the outcome. Only the changes over time without the policy must match.

Parallel trends depends on how the outcome is measured. On this page, the treated stores start at a higher level. If both groups would gain the same amount of sales, the treated stores would gain a smaller percentage. Then the trends are parallel in sales but not in percentages or in logarithms. So a study must explain why the assumption holds on the scale that it uses (Cunningham 2021, ch. 9).

The policy must also leave the comparison group unaffected. Suppose that the policy also changes sales in the comparison stores. Then their change no longer shows what would have happened without the policy (Huntington-Klein 2025, ch. 18).

Suppose that parallel trends holds and that the policy has no effect before it starts (no anticipation). Then DiD estimates the average effect of the policy on the stores that got it, in the periods after the start. This quantity is the average treatment effect on the treated (ATT). DiD says nothing about the effect that the policy would have on the comparison group (Huntington-Klein 2025, ch. 18).

Assume parallel trends, no anticipation, and no spillovers. Then, in symbols, the DiD calculation equals the ATT:

\[\text{ATT} = E[Y_{\text{after}}(1) - Y_{\text{after}}(0) \mid \text{treated}]\]

In general, DiD equals the ATT plus a bias term. The bias term is the change in \(Y(0)\) for the treated group minus the change in \(Y(0)\) for the comparison group. Under parallel trends, this term is zero (Cunningham 2021, ch. 9).

Parallel trends is an assumption about outcomes that nobody observes: the outcomes of the treated group after the start date without the policy. So the data cannot test it directly. Parallel lines before the start date make the assumption more believable. They do not prove it (Cunningham 2021, ch. 9).

Lines that move apart before the start date do not prove that the assumption fails, either. The gap could stop growing at the start date for reasons other than the policy. Then parallel trends would still hold (Huntington-Klein 2025, ch. 18).

An example comes from a study of the minimum wage (Card and Krueger 1994). On April 1, 1992, the minimum wage in New Jersey rose from $4.25 to $5.05 per hour. The minimum wage in Pennsylvania did not change. The authors measured employment at fast-food restaurants in New Jersey and in eastern Pennsylvania before and after the rise. The Pennsylvania restaurants were the comparison group.