5.2 Difference-in-differences
Staggered adoption: when two-way fixed effects mislead
In staggered adoption, groups of units start a policy in different periods. This page shows when the usual regression for this design gives a misleading average effect. It also shows a method that first estimates a separate effect for each group and each period.
In brief
A retail chain starts a new policy in its stores in waves. An early group of stores adopts the policy in period 4, and a late group adopts it in period 7. A third group, the never group, never adopts it.
- A two-way fixed effects (TWFE) regression has a separate intercept for each store and for each period, and a treatment indicator for the policy. The coefficient on this indicator is the TWFE estimate.
- The true average effect over all treated store-periods averages the effect over every store in every period with the policy. Only the simulation knows it.
- Each comparison in the decomposition chart is a difference-in-differences with two groups and two windows of time. The TWFE estimate is a weighted average of these comparisons.
- The Callaway and Sant'Anna method first estimates the average effect of each group in each period, the group-time effect. It compares each group only with stores that do not have the policy. The Callaway and Sant'Anna estimate averages these effects, weighted by group size.
Move the sliders and watch these numbers. The explanation below the simulation gives the reasons.
Try it
The simulation has 40 stores in each group. The sliders set the effect in the first period, its change per period, an extra effect in the early group, and the noise. A box can remove the stores that never adopt.
How the simulation makes the data
Each store has its own level of the outcome. The outcome of each store also moves up and down at random from period to period. This random part is the noise.
Without the policy and apart from the noise, all stores have the same changes from one period to the next. So parallel trends holds for every pair of groups. Any bias on this page comes from the way an estimator uses the data, not from a failure of parallel trends.
The simulation also keeps the outcome that each store would have without the policy, so it knows every true effect. The early group has 280 treated store-periods (40 stores in periods 4 to 10). The late group has 160 (40 stores in periods 7 to 10).
- TWFE estimate
- True average effect over all treated store-periods
- TWFE estimate minus true effect
- Callaway and Sant'Anna estimate (simple weighted average)
- True average effect over the store-periods that the method can estimate
- Callaway and Sant'Anna estimate minus true effect
The page compares the TWFE estimate with the true average effect over all treated store-periods. It compares the Callaway and Sant'Anna estimate with the true average effect over the store-periods that this method can estimate. In one sample, each difference is the bias plus the sampling error. Draw new samples, or set the noise to 0, to see the bias. The key terms on the home page define bias and sampling error. The page rounds each number to two decimals, so a difference of two printed numbers can be off by 0.01.
Inside the TWFE estimate: the decomposition chart
| Comparison | Periods used | Weight | Estimate | Weight × estimate |
|---|
In the sum row, the weights add to 1, and the last column adds to the TWFE estimate. The table rounds each number. So a sum or a product of printed numbers can differ from the printed result in the last digit.
Effects by periods since adoption
| Periods since adoption | Groups in the average | Callaway and Sant'Anna estimate | True value |
|---|
The chart of effects by periods since adoption shows only periods after adoption. Each group-time effect in it measures the change from period \(g - 1\), the last period before adoption (Callaway and Sant'Anna 2021). We call period \(g - 1\) the base period. Software for group-time estimators can also show periods before adoption. Those estimates can use a different base period than the estimates after adoption. So check which base period a program uses before you compare its chart with an event-study chart, such as the chart in Module 5.3.
The idea
A retail chain starts a new policy in its stores in waves. We say that a store adopts the policy when it starts the policy. An early group of stores adopts the policy in period 4. A late group adopts it in period 7. A third group never adopts it. We call this group the never group.
Once a store has the policy, it keeps it. Huntington-Klein (2025, ch. 18) calls this setting a rollout design.
The usual estimate comes from a two-way fixed effects (TWFE) regression. The regression has a separate intercept for each store (store fixed effects) and for each period (period fixed effects). It also has a treatment indicator, D. D is 1 for a store in a period with the policy and 0 otherwise. The coefficient on D is the TWFE estimate.
The store fixed effects absorb every difference between stores that does not change over time, for example a difference in location or in management style. The period fixed effects absorb every change that hits all stores by the same amount in the same period. The season is one example (Huntington-Klein 2025, ch. 16).
Goodman-Bacon (2021, Theorem 1) shows that the TWFE estimate is a weighted average of simple comparisons. Each comparison is a difference-in-differences with two groups and two windows of time. It takes the change in the mean outcome of a group that adopts between the two windows. Then it subtracts the same change in a comparison group, a group whose policy status does not change between the windows. Many texts call this group the control group.
With three groups there are four comparisons. Each label has the form A vs B (A versus B). A is the group that adopts, and B is the comparison group:
- Early vs never uses periods 1 to 3 and 4 to 10.
- Late vs never uses periods 1 to 6 and 7 to 10.
- Early vs late uses periods 1 to 3 and 4 to 6. In these periods, the late group does not have the policy yet.
- Late vs early uses periods 4 to 6 and 7 to 10. In these periods, the early group already has the policy.
The same group can play both roles. In early vs late, the late group is the comparison group. In late vs early, the late group is the group that adopts.
The comparisons in symbols
Let \(T\) be the last period (here \(T = 10\)). Let \(a_k\) be the first period with the policy in group \(k\), and let \(U\) be the never group. The decomposition assumes a balanced panel, with data on every store in every period. It also assumes that no group has the policy in period 1 (Goodman-Bacon 2021, Theorem 1).
Write \(\bar{y}_j(W)\) for the mean outcome of group \(j\) over the periods in window \(W\). \(\text{PRE}(k)\) is the periods before \(a_k\). \(\text{POST}(k)\) is the periods \(a_k\) to \(T\). For an earlier group \(k\) and a later group \(l\), \(\text{MID}(k,l)\) is the periods \(a_k\) to \(a_l - 1\). The three kinds of comparison are:
\[\begin{aligned} b_{kU} = {} & [\bar{y}_k(\text{POST}(k)) - \bar{y}_k(\text{PRE}(k))] \\ & - [\bar{y}_U(\text{POST}(k)) - \bar{y}_U(\text{PRE}(k))] \\ b^k_{kl} = {} & [\bar{y}_k(\text{MID}(k,l)) - \bar{y}_k(\text{PRE}(k))] \\ & - [\bar{y}_l(\text{MID}(k,l)) - \bar{y}_l(\text{PRE}(k))] \\ b^l_{kl} = {} & [\bar{y}_l(\text{POST}(l)) - \bar{y}_l(\text{MID}(k,l))] \\ & - [\bar{y}_k(\text{POST}(l)) - \bar{y}_k(\text{MID}(k,l))] \end{aligned}\]
The weights are never negative, and they add up to 1. They depend on the sizes of the groups and on when each group adopts. The panel is the data on all stores in all periods, here periods 1 to 10. Groups that adopt near the middle of the panel get more weight (Goodman-Bacon 2021).
The weights in symbols
Let \(n_k\) be the share of stores in group \(k\). Let \(\bar{D}_k = (T - a_k + 1)/T\) be the share of periods in which group \(k\) has the policy. This share is the mean of \(D\) in group \(k\). Let \(V\) be the mean of the squared values of \(D\) after we remove the store means and the period means from \(D\). After some algebra, the weights in Goodman-Bacon (2021, Theorem 1) are:
\[\begin{aligned} s_{kU} &= \frac{n_k n_U \bar{D}_k(1 - \bar{D}_k)}{V} \\ s^k_{kl} &= \frac{n_k n_l (\bar{D}_k - \bar{D}_l)(1 - \bar{D}_k)}{V} \\ s^l_{kl} &= \frac{n_k n_l \bar{D}_l(\bar{D}_k - \bar{D}_l)}{V} \end{aligned}\]
The TWFE estimate is the sum of each weight times its comparison.
The problem is in the last kind of comparison. Its comparison group already has the policy. Suppose that the effect of the policy grows over time. Then the outcome of the comparison group rises for that reason too. The comparison subtracts this rise from the effect of the late group. So an effect that changes over time biases this comparison, and through it the TWFE estimate (Goodman-Bacon 2021).
Huntington-Klein (2025, ch. 18) puts the problem this way: the regression uses groups that already have the policy as a comparison group.
Effects with a minus sign, and negative weights
No comparison has a negative weight. Yet when effects change over time, some effects still enter the TWFE estimate with a minus sign. The reason lies in each comparison whose comparison group already has the policy. Such a comparison subtracts the change in the effect of that group between the two windows (Goodman-Bacon 2021).
In symbols, write \(\text{ATT}_j(W)\) for the average effect in group \(j\) over the periods in window \(W\). If parallel trends holds, the comparison \(b^l_{kl}\) (late vs early on this page) estimates:
\[\begin{aligned} & \text{ATT}_l(\text{POST}(l)) \\ & - [\text{ATT}_k(\text{POST}(l)) - \text{ATT}_k(\text{MID}(k,l))] \end{aligned}\]
The change in the effect of group \(k\) enters with a minus sign (Goodman-Bacon 2021).
De Chaisemartin and D'Haultfœuille (2020) write the TWFE estimate in another way. They write it as a weighted sum of the effects in each group and period with the policy. Some of these weights can be negative. Then the TWFE estimate can be negative when every effect is positive. They give such an example with two groups and three periods.
The weights of de Chaisemartin and D'Haultfœuille (2020) are weights on the effects in each group and period with the policy, not on the comparisons. So their weights are not the weights of the decomposition above.
When the effect does not change over time
A staggered design alone does not make TWFE biased. If the effect is one constant number, every comparison estimates that number, and so does TWFE. If the effect differs between groups but does not change over time, no comparison is biased. TWFE is then a weighted average of the group effects. These weights come from the weights of the comparisons. So they also give more weight to groups that adopt near the middle of the panel (Goodman-Bacon 2021).
The average effect over all treated store-periods weights the groups in another way. Treated store-periods are the stores in the periods when they have the policy. This average weights each group by its number of treated store-periods. So it gives more weight to groups that adopt early, because these groups have the policy for more periods. For this reason, TWFE can differ from the average effect over all treated store-periods.
Callaway and Sant'Anna (2021) do not use stores that already have the policy as a comparison group. They first estimate a group-time average treatment effect, written ATT(g,t). Here ATT stands for average treatment effect on the treated (see the key terms). ATT(g,t) is the average effect in period t for the stores that adopt in period g.
The group-time effect in symbols
For Callaway and Sant'Anna (2021), let \(G\) be the first period with the policy for a store, and let \(g\) be one value of \(G\). So \(g\) plays the role of \(a_k\) above. Write \(Y_t(1)\) and \(Y_t(0)\) for the outcomes of a store in period \(t\) with and without the policy. Write \(Y_t\) for its observed outcome. \(E[\,\cdot\,]\) is an average over the population of stores, and a vertical bar means among.
The group-time effect is \(\text{ATT}(g,t) = E[Y_t(1) - Y_t(0) \mid G = g]\).
Period \(g - 1\) is the last period before group g adopts. To estimate ATT(g,t), the method takes the change in the mean outcome of group g from period \(g - 1\) to period t. Then it subtracts the same change in a comparison group that does not have the policy in either period. On this page, the comparison group is the never group. Without a never group, the comparison group is the stores that have not adopted by period t.
The method needs two assumptions. First, stores do not react to the policy before they adopt it. Second, the parallel trends assumption holds for group g and its comparison group. That is, without the policy, both mean outcomes would have the same change from period \(g - 1\) to period t (Callaway and Sant'Anna 2021).
In symbols, with these two assumptions and no covariates, the group-time effect equals a difference of observed changes:
\[\begin{aligned} \text{ATT}(g,t) = {} & E[Y_t - Y_{g-1} \mid G = g] \\ & - E[Y_t - Y_{g-1} \mid \text{comparison group}] \end{aligned}\]
The comparison group is the never group, or the stores with \(G > t\) if there is no never group.
Without a never group
Without a never group, the method runs out of stores for the comparison group. From period 7 on, every store has the policy. So the method can estimate the effects of the early group only in periods 4 to 6. It cannot estimate any effect of the late group (Callaway and Sant'Anna 2021). The decomposition of TWFE does not need a never group (Goodman-Bacon 2021).
Next, the method averages the group-time effects. This page shows the simple weighted average of the group-time effects. It weights each group-time effect by the size of its group. So it is the average effect over the treated store-periods that the method can estimate.
In symbols, the simple weighted average gives each \(\text{ATT}(g,t)\) with \(t \ge g\) a weight proportional to \(P(G = g)\). Without a never group, this page uses only the \(\text{ATT}(g,t)\) that the method can estimate.
Other averages of the group-time effects
The paper also proposes other summaries, including one that first averages within each group (Callaway and Sant'Anna 2021). The page also averages the group-time effects by the number of periods since adoption. The set of groups in that average changes with the number of periods.
In symbols, the average by periods since adoption, \(e\), is:
\[\begin{aligned} & \theta_{\text{es}}(e) = \sum_g 1\{g + e \le T\} \\ & \qquad P(G = g \mid G + e \le T) \, \text{ATT}(g, g + e) \end{aligned}\]
Here \(1\{\cdot\}\) is 1 when the condition holds and 0 otherwise.
Before you report a TWFE estimate from a staggered design, check it in two steps. First, split the estimate into its comparisons, as the decomposition chart above does. Look at how much weight goes to comparisons whose comparison group already has the policy. Look also at how far their estimates are from the other estimates (Goodman-Bacon 2021). Second, estimate the group-time effects and average them (Callaway and Sant'Anna 2021).
A large gap between the TWFE estimate and this average means that the comparisons inside TWFE disagree. One TWFE number can hide comparisons that point in opposite directions. So report the group-time effects and their average.
References
- Callaway, B., & Sant'Anna, P. H. C. (2021). Difference-in-differences with multiple time periods. Journal of Econometrics, 225(2), 200-230. https://doi.org/10.1016/j.jeconom.2020.12.001
- de Chaisemartin, C., & D'Haultfœuille, X. (2020). Two-way fixed effects estimators with heterogeneous treatment effects. American Economic Review, 110(9), 2964-2996. https://doi.org/10.1257/aer.20181169
- Goodman-Bacon, A. (2021). Difference-in-differences with variation in treatment timing. Journal of Econometrics, 225(2), 254-277. https://doi.org/10.1016/j.jeconom.2021.03.014
- Huntington-Klein, N. (2025). The effect: An introduction to research design and causality (2nd ed.). Chapman & Hall. Chapter 16, Fixed Effects, and chapter 18, Difference-in-Differences. Free online edition: chapter 16 and chapter 18.