3.1 Noncompliance

Noncompliance and instrumental variables (IV): what a randomized ad can estimate

An online store can randomize which ad each customer gets, but each customer decides whether to engage with it. This page uses a field experiment with a swipe ad to show what such an experiment can estimate. It also shows why comparisons by the customers' own choices mislead.

In brief

A random half of the customers get a swipe ad that hides the book until the end: the book appears only after the last frame. The other customers get a standard ad that shows the book at once. A customer completes the swipe ad by swiping through every frame to this reveal, and few customers do. This page asks what completion does to spending.

The intention-to-treat (ITT) effect is the difference in average spending between the two assigned groups.
The first stage is the difference between the two groups in the share who complete.
The Wald estimate is the ITT effect divided by the first stage.
Under the assumptions in the explanation, the Wald estimate estimates the local average treatment effect (LATE). This is the average effect of completion for compliers, the customers who complete only if they get the swipe ad.

Move the sliders and watch these numbers. The explanation below the simulation gives the reasons.

Try it

The simulation starts with 2,000 customers, and half of them get the swipe ad. The sliders set the shares of the compliance types, the effect of completion, and two possible problems. In the study, nobody with the standard ad could complete, so the sliders for always-takers and defiers start at 0.

How the simulation makes the data

Each customer gets a type, with the shares on the sliders. The rest are never-takers. Without the swipe ad, compliers and defiers spend about 40 dollars on books in 30 days on average. Always-takers spend more by the amount on the gap slider. Never-takers spend less by the same amount.

Completion adds the complier effect, the LATE, to the spending of compliers. For always-takers, it adds the complier effect plus 5 dollars. For never-takers, it adds the complier effect minus 5 dollars. For defiers, it adds the complier effect minus 10 dollars.

The simulation knows the type and both potential outcomes of every customer. Real data show only the assignment, the completion, and the spending. All amounts on this page are simulated, in dollars.

Who is in each observed group

Each column counts the customers with one combination of assignment and completion. Real data show the height of each column but not its colors. The colors come from the simulation, which knows each customer's type. Hover over a part to see its count.

Four estimates in this sample

Each dot is an estimate, and each vertical line is its 95% confidence interval. The dark dashed line marks the true LATE, the effect for compliers. The light dashed line marks the true ATE, the average effect for all customers. Hover over a dot to see its values.

The Wald estimate over 2,000 repeated experiments

The page repeats the experiment 2,000 times with new customers and the same settings. The bars count the Wald estimates. The dashed line marks the true LATE. The orange line marks the average as-treated comparison. Estimates outside the range of the horizontal axis are counted in the readout below. Hover over a bar to see its count.

The compliance types (simulation only)
Type	Share in this sample	Spending without the swipe ad	Effect of completion
Compliers: complete only with the swipe ad
Always-takers: complete with either assignment
Never-takers: never complete
Defiers: complete only with the standard ad

The four estimates in this sample
Estimate	Value	95% confidence interval
A. Intention-to-treat effect on spending
B. As-treated comparison
C. Per-protocol comparison
D. Wald estimate

Share who complete: with the swipe ad and with the standard ad
First stage (difference in the share who complete)
First-stage F statistic
ITT effect on spending (95% interval)
Wald estimate (95% interval)
Anderson-Rubin 95% interval
True LATE (simulation only)
True ATE (simulation only)
Wald estimate minus true LATE
Expected part of that difference from the direct effect
Expected part of that difference from defiers

The two expected parts come from the error formula in the box on two-sided noncompliance below, with the shares on the sliders. In one sample, the Wald estimate also moves by chance. The readouts below summarize the 2,000 repeated experiments.

Median of the Wald estimates
Median first-stage F statistic
Usual 95% intervals that contain the true LATE
Anderson-Rubin intervals that contain the true LATE
Anderson-Rubin intervals that are unbounded
Average as-treated comparison
Wald estimates outside the chart
Experiments with no Wald estimate (first stage of exactly 0)

The usual interval is the Wald estimate plus or minus 1.96 times its standard error. Its coverage leaves out the experiments with no Wald estimate. The ATE, the LATE, the p-value, and other terms are defined in the key terms on the home page.

What to notice

At the default settings, half of the customers are compliers, and nobody completes with the standard ad. The ITT effect on spending is 5.06, about half of the true LATE of 10. The Wald estimate divides it by the first stage of 0.502 and gives 10.09. The as-treated comparison (16.55) and the per-protocol comparison (14.93) are too high, because never-takers, who spend less, fill the group that did not complete. Set the gap to 0: both comparisons fall to about 10. The light dashed line, the ATE of 7.54, stays below the LATE, because never-takers would gain less from completion.
Set the gap back to 10 and the complier share to 3%, close to the study. The ITT effect falls to 0.57, and its interval, from −0.31 to 1.45, contains 0. The first stage is only 0.032, so the Wald estimate of 17.92 has a wide interval, from −8.84 to 44.68. In the repeated experiments, the median Wald estimate is 9.53, close to the true LATE, and the median first-stage F statistic is 30.9.
Keep the complier share at 3% and set the direct effect to 1. The expected part from the direct effect is 32.33: the direct effect times the never-taker share (97%) divided by the complier share (3%). The Wald estimate jumps to 48.17, and only 46.2% of the usual intervals in the repeated experiments contain the true LATE.
Set the direct effect back to 0, the complier share back to 50%, and the share of defiers to 10%. Some customers now complete only with the standard ad, which the study design rules out. The Wald estimate is 12.78, above both the complier effect (10) and the defier effect (0). The expected part from defiers is 2.50.
Set the defiers back to 0, the complier share to 3%, the always-takers to 20%, and the gap to 30. The assignment now moves few customers compared with the noise. In Sample 1 the first-stage F statistic is 0.5, and the Wald estimate is −51.66 with an interval from −404.16 to 300.83. In the repeated experiments, the median Wald estimate is 15.77, pulled toward the as-treated comparison near 72. Only 87.8% of the usual intervals contain the true LATE, while 95.3% of the Anderson-Rubin intervals do. Most Anderson-Rubin intervals (62.4%) are unbounded.

The idea

An online bookstore wants to know whether a new kind of ad sells books. The ad is an interactive blind lead-in ad: a swipe ad that reveals the book only at the end, after its last frame. The frames themselves do not show the book. A customer completes the ad by swiping through every frame to the reveal.

A field experiment tested this ad with more than 109,000 customers of a large online bookstore in South Korea (Yoon et al. in press). It assigned each customer at random to the swipe ad or to a standard ad that shows the book at once. Only about 3 in 100 exposures to the swipe ad reached the reveal. On average, assignment to the swipe ad did not raise purchases. An instrumental variables analysis found that completion raised later purchases among the customers whom the swipe ad moved to complete it. This page shows how one experiment can give both results.

The assignment, Z, is random. It is an instrumental variable, or instrument, for completion: it is assigned at random, and it changes who completes. Completion, D, is the treatment of interest, and each customer chooses it. A customer who gets the swipe ad and stops early does not take the treatment that the assignment offers. This is called noncompliance.

The outcome, Y, is spending on books in the next 30 days. The study measured purchases. This page simulates spending, so that the numbers are easy to read.

Each customer has a compliance type, defined by the choices that the customer would make with each assignment (Angrist, Imbens, and Rubin 1996). A customer with the standard ad has no reveal to reach. So the experiment has only two types. Compliers would complete the swipe ad if they got it. Never-takers would not complete it.

This setting is called one-sided noncompliance (Imbens and Rubin 2015, ch. 23). The assignment cannot change a customer's type, so the types are principal strata, as in Module 3.2. The data show the type only for the customers who got the swipe ad.

The types in symbols

\(D(z)\) is the completion of a customer with assignment \(z\) (1 for the swipe ad, 0 for the standard ad). \(Y(z, d)\) is spending with assignment \(z\) and completion \(d\). The exclusion restriction says that \(Y(z, d)\) does not depend on \(z\), so it can be written \(Y(d)\).

The types are compliers (\(D(0) = 0, D(1) = 1\)), always-takers (1, 1), never-takers (0, 0), and defiers (1, 0). Their shares are \(\pi_c\), \(\pi_a\), \(\pi_n\), and \(\pi_d\). One-sided noncompliance means \(D(0) = 0\) for every customer, so \(\pi_a = \pi_d = 0\).

Three simple comparisons are common. The intention-to-treat (ITT) effect compares the average spending of the two assigned groups. Random assignment makes this comparison unbiased for the effect of the assignment. The ITT effect is not the effect of completion. It averages over compliers and never-takers, so with 3 compliers in 100 customers even a large effect of completion gives a small ITT effect.

The as-treated comparison puts the customers who completed against all customers who did not. The per-protocol comparison keeps only the customers who followed their assignment. It compares the customers who completed the swipe ad with the customers who got the standard ad. Both comparisons use groups that the customers formed by their own choices. Customers who swipe through to the reveal may be keen readers who buy more books anyway. So these comparisons bring back the selection bias that random assignment removed (Imbens and Rubin 2015, ch. 23).

The Wald estimator divides the ITT effect on spending by the ITT effect on completion. The second number, the difference in the share who complete, is the first stage.

\[\begin{aligned} \text{Wald} &= \frac{\text{ITT}_Y}{\text{ITT}_D} \\ &= \frac{E[Y \mid Z = 1] - E[Y \mid Z = 0]}{E[D \mid Z = 1] - E[D \mid Z = 0]} \end{aligned}\]

The Wald estimator estimates the local average treatment effect (LATE), the average effect of completion for compliers. It does so if these assumptions hold (Angrist, Imbens, and Rubin 1996; Huntington-Klein 2025, ch. 19):

The assignment is random.
The assignment changes spending only through completion. This is the exclusion restriction.
The assignment changes who completes, so the first stage is not zero.
No customer completes only with the standard ad. Such a customer would be a defier, and this condition is called monotonicity. One-sided noncompliance makes it hold.
The assignment of one customer does not change the choices or the spending of another customer. This is the no-interference condition in the key terms.

Under these assumptions, the ITT effect on spending equals the complier share times the LATE. With one-sided noncompliance, every customer who completed is a complier with the swipe ad. Random assignment makes these customers like all compliers. So the Wald estimator then also estimates the average effect for the customers who completed (Imbens and Rubin 2015, ch. 23).

\[\begin{aligned} \text{ITT}_D &= \pi_c, \qquad \text{ITT}_Y = \pi_c \times \text{LATE}, \\ \text{Wald} &= \frac{\pi_c \times \text{LATE}}{\pi_c} = \text{LATE} \end{aligned}\]

The usual regression method for instrumental variables is two-stage least squares (2SLS). With one random assignment and no other variables, it gives the same number as the Wald estimator. The LATE says nothing about never-takers. The average treatment effect (ATE) for all customers can be different.

The exclusion restriction needs thought in a design like this one. A never-taker with the swipe ad saw some frames but not the book. A never-taker with the standard ad saw the book at once. If this difference changes spending, the assignment has a direct effect on spending, and the exclusion restriction fails.

The Wald estimator then misses the LATE. Without defiers, the error is the direct effect times the share of never-takers, divided by the share of compliers (Angrist, Imbens, and Rubin 1996):

\[\text{Wald} - \text{LATE} = h \, \frac{\pi_n}{\pi_c}\]

Here \(h\) is the direct effect of the swipe ad on the customers who get it and do not complete. With 3 compliers in 100, a direct effect of 1 dollar moves the Wald estimate by about 32 dollars.

Weak instruments and the Anderson-Rubin interval

A small first stage makes the Wald estimate imprecise, because the sampling error of the ITT effect is divided by a small number. A weak instrument moves so few people that the first stage is small compared with its own sampling error. Then the Wald estimates spread out, and their median moves toward the as-treated comparison (Huntington-Klein 2025, ch. 19). The usual 95% interval also contains the truth less often than 95% of the time. The size of these problems also depends on how closely completion is related to spending without the swipe ad (Andrews, Stock, and Sun 2019).

The usual check is the F statistic of the first stage: the square of the first stage divided by its standard error. A common rough rule asks for an F above 10. The Anderson-Rubin (AR) interval stays valid with a weak instrument. For each candidate effect b, it subtracts b times completion from spending. Then it tests whether the two assigned groups differ in the result. The interval keeps every b that this test does not reject at the 5% level, as Module 1.1 does for a constant effect.

With a weak instrument, the AR interval can be unbounded, for example two half-lines (Andrews, Stock, and Sun 2019).

In symbols, the AR interval collects every value \(b\) for which a test does not reject equal means of \(Y - bD\) in the two assigned groups. The test uses the 5% level.

Two-sided noncompliance: always-takers and defiers

In other experiments, noncompliance is two-sided. Some people in the comparison group take the treatment anyway, for example the same ad from another source. They are always-takers. In principle, some people could do the opposite of their assignment. They are defiers.

Suppose that defiers exist, that compliers outnumber them, and that their effect differs from the LATE. Then the Wald estimator targets a value outside the range between the two effects, beyond the LATE (Imbens and Rubin 2015, ch. 24). The sliders for these two types show both cases. In the study, both shares are 0.

In symbols, let \(\tau_c\) and \(\tau_d\) be the average effects of completion for compliers and defiers. The first stage is \(\text{ITT}_D = \pi_c - \pi_d\). In the simulation on this page,

\[\text{Wald} - \tau_c = \frac{h(\pi_n + \pi_d) + \pi_d(\tau_c - \tau_d)}{\pi_c - \pi_d}\]

References

Andrews, I., Stock, J. H., & Sun, L. (2019). Weak instruments in instrumental variables regression: Theory and practice. Annual Review of Economics, 11(1), 727-753. https://doi.org/10.1146/annurev-economics-080218-025643
Angrist, J. D., Imbens, G. W., & Rubin, D. B. (1996). Identification of causal effects using instrumental variables. Journal of the American Statistical Association, 91(434), 444-455. https://doi.org/10.1080/01621459.1996.10476902
Huntington-Klein, N. (2025). The effect: An introduction to research design and causality (2nd ed.). Chapman & Hall. Chapter 19, Instrumental Variables. Free online edition: https://theeffectbook.net/ch-InstrumentalVariables.html
Imbens, G. W., & Rubin, D. B. (2015). Causal inference for statistics, social, and biomedical sciences: An introduction. Cambridge University Press. Chapter 23, Instrumental Variables Analysis of Randomized Experiments with One-Sided Noncompliance, and chapter 24, Instrumental Variables Analysis of Randomized Experiments with Two-Sided Noncompliance. https://doi.org/10.1017/CBO9781139025751
Yoon, S., Yoo, J., Kim, M., Kim, W., & Lee, M. S. (in press). Interactive blind lead-in ads and purchase conversion: The roles of completion and creative design features. Journal of Interactive Marketing. The study used as the example on this page. Abstract: https://j1yoo.github.io/research/