Regression adjustment and good and bad controls: what to adjust for, and how

In brief

A shop sells a paid membership, and each customer decides whether to join. The shop wants the effect of joining on spending in the next month. Loyal customers join more often, and they also spend more without the membership.

• The average treatment effect (ATE) is the average effect of joining over all customers. The average treatment effect on the treated (ATT) is the average effect over the members.
• Estimate A, the simple difference in means, mixes the effect with an old gap between members and nonmembers.
• Estimate C, the segment differences weighted by segment size, compares members with nonmembers inside each segment. Under the assumptions in the explanation, C estimates the ATE.
• Estimate D, the regression with a separate intercept for each segment, gives little weight to segments where almost everyone or almost nobody joins.
• In part 2, a chain of stores wants the effect of local advertising on sales. The table shows the average estimate of two regressions over repeated samples, without and with a third variable, Z.

Move the sliders and watch these numbers. The explanation below the simulation gives the reasons.

The idea

A shop sells a paid membership, and each customer decides whether to join. The shop wants the effect of joining on spending in the next month. Loyal customers join more often, and they also spend more without the membership. So the simple difference between members and nonmembers mixes the effect with an old gap. It equals the average treatment effect on the treated (ATT) plus selection bias, as the key terms explain.

Suppose that the customer segment (new, occasional, regular, or loyal) is the only common cause of joining and spending. This is the assumption of no unmeasured confounding, also called unconfoundedness or selection on observables. Suppose also that every segment has both members and nonmembers. This is called overlap, or positivity. Then a comparison inside each segment is unbiased.

Comparing inside segments is called adjusting for the segment, and a variable that a study adjusts for is called a control. The average of the comparisons, weighted by the segment shares of all customers, is the average treatment effect (ATE). This method is regression adjustment by imputation. It predicts the spending of each customer with and without the membership, here from the segment averages. Then it averages the difference over all customers (Hernán and Robins 2020, ch. 13).

A common shortcut is one regression of spending on a membership indicator and a separate intercept for each segment. Its single membership coefficient weights each segment by its size times \(p(1 - p)\), where p is the segment's share of members (Angrist 1998). Segments where almost everyone or almost nobody joins get little weight. If the effect differs across segments, this coefficient is a different average than the ATE. Module 1.2 shows the same weights in a block experiment.

In symbols, let the segments be \(s = 1, \ldots, 4,\) with segment shares \(w_s\). Let \(p_s\) be the member share and \(\tau_s\) the effect in segment \(s\). The imputation estimator targets \(\sum w_s \tau_s = \text{ATE}\). The membership coefficient of the regression with segment intercepts targets

\[\frac{\sum w_s p_s (1 - p_s) \tau_s}{\sum w_s p_s (1 - p_s)}\]

Which variables belong in the adjustment? A causal diagram, or directed acyclic graph (DAG), draws each variable as a node and each direct cause as an arrow. A path is a chain of arrows between two variables, in any direction. Adjusting for a variable on a path blocks the path, unless the variable is a collider.

A back-door path is a path between the treatment and the outcome that starts with an arrow into the treatment. Adjusting for a set of variables gives the causal effect if the set blocks every back-door path and contains no variable that the treatment affects. This rule is the back-door criterion (Cinelli, Forney, and Pearl 2024).

The rule sorts candidate variables into good and bad controls (Cinelli, Forney, and Pearl 2024):

• A confounder, a common cause of the treatment and the outcome, is a good control.
• A mediator lies on the path from the treatment to the outcome. The total effect of the treatment includes the part that runs through the mediator. So for the total effect, a mediator is a bad control: adjusting for it removes part of the effect.
• A collider is a common effect of two variables on a path. The collider blocks the path by itself, and adjusting for it opens the path (Huntington-Klein 2025, ch. 8). The bias that this creates is called collider bias. Keeping only the stores that won an award for high advertising or high sales works the same way, because the award is a collider.
• A variable measured before the treatment can still be a collider. In the M-structure, it has two hidden causes, which nobody measured. One of them also causes the treatment, and the other also causes the outcome. In the simulation above, the bias from adjusting for it stays small unless the arrows are strong.
• A cause of the outcome only does not change the bias, and adjusting for it usually makes the estimate more precise.
• A cause of the treatment only does not change the bias either, but adjusting for it makes the estimate less precise. If a hidden confounder also exists, adjusting for the cause of the treatment can increase the bias. In linear models, this adjustment always increases the bias.

In part 2, every variable is a linear function of its causes plus independent noise with variance 1. Advertising D has a true effect of 1 on sales Y. The strength slider sets every arrow into and out of Z, and in the M-structure also the arrows from the hidden variables. In the last case, the hidden confounder keeps a strength of 0.6.