1.2 Randomized experiments

Neyman variance and blocking: how much an estimate moves when the coin flips change

In an experiment, the estimate depends on which units the coin flips put in the treated group. This page shows how much the estimate moves from one set of coin flips to another. It also shows why the Neyman standard error overstates this movement when effects differ across units, and when blocking makes the estimate more precise.

In brief

A shop sends a discount offer to a random half of its customers. A computer draws which half gets the offer, and this page calls this draw the coin flips. The difference between the mean spending of the two groups estimates the effect of the offer.

The sample average treatment effect (SATE) is the average effect among the customers in the experiment. The population average treatment effect (PATE) is the average effect in the larger population that the customers come from.
The true standard deviation of the estimate measures how much the estimate moves from one experiment to the next. Only a simulation knows it.
The Neyman standard error estimates this movement from the data of one experiment. It equals the robust standard error HC2 (heteroskedasticity-consistent standard error of type 2) of a regression.
The classical standard error of a regression assumes that the outcome has the same variance in both groups.
In Part 2, blocking runs the coin flips inside each customer segment. Estimator C averages the segment estimates, weighted by segment size.

Move the sliders and watch these numbers. The explanation below the simulation gives the reasons.

Try it

The simulation draws customers with both potential outcomes. Part 1 repeats the coin flips on the same customers. Part 2 sorts the customers into four segments and compares two designs.

How the simulation makes the data

In Part 1, spending without the offer is drawn around 40 dollars with a standard deviation of 10. The offer adds an individual effect to each customer's spending. The effects average 10 dollars in the population and differ across customers by the spread slider. All amounts on this page are in dollars.

Part 1. The same customers, new coin flips

The page repeats the experiment 1,000 times. Each time, a new set of coin flips decides who gets the offer, and the page computes the estimate and both standard errors.

The bars count the estimates from the 1,000 repeated experiments, and real data show only one of them. The three curves are normal (bell-shaped) curves centered on the true target, which the dashed vertical line marks. The dashed curve uses the true standard deviation of the estimate, and the two solid curves use the average of each standard error. A curve that is wider than the bars belongs to a standard error that is too large. At a share of 50%, the two standard errors are equal, so the orange curve covers the green one. Hover over a bar to see its count.

True SATE
Average of the 1,000 estimates
True standard deviation of the estimate (simulation only)
Average Neyman standard error (equal to HC2)
Average classical standard error
95% intervals with the Neyman standard error that contain the target
95% intervals with the classical standard error that contain the target
Customers with and without the offer

Parts of the true variance of the estimate (simulation only)
Part	Value
S1² / n1
S0² / n0
Variance of the individual effects divided by n (subtracted)
True variance of the estimate
Average of the Neyman variance estimate over all possible experiments

The variances are in squared dollars. The square root of the true variance is the true standard deviation in the readout. When the customers also change (target: PATE), the table does not subtract the variance of the individual effects. Then the true variance equals the average of the Neyman variance estimate. The table rounds each number to two decimals. The SATE, the 95% confidence interval, and other terms are defined in the key terms on the home page.

Part 2. Blocking by customer segment

The shop has four segments of the same size: new, occasional, regular, and loyal customers. Average spending without the offer rises from one segment to the next by the gap slider. The page compares two designs on the same customers. Complete randomization gives the offer to the same total number of customers, drawn from all segments together. Block randomization draws the treated customers inside each segment. The page repeats each design 2,000 times.

Each dot is the average of the estimates from one estimator over the 2,000 repeated experiments. Each vertical line covers the middle 95% of its estimates. A shorter line means a more precise estimator. A dot away from the dashed line means a biased estimator. Hover over a dot to see its values.

The four estimators over 2,000 repeated experiments
Design and estimator	Average estimate	Average minus true SATE	Standard deviation
A. Complete randomization: difference in means
B. Block randomization: difference in means that ignores the segments
C. Block randomization: segment estimates weighted by segment size
D. Block randomization: regression with segment fixed effects

The segments and the weights of estimators C and D
Segment	New	Occasional	Regular	Loyal
Treated customers
Spending without the offer
True effect
Weight in C
Weight in D

Each segment has the same number of customers. The spending without the offer is the segment average that the simulation uses, and the true effect is the effect that the simulation adds. Only a simulation knows them. Estimator C weights each segment by its share of the customers. Estimator D weights it by its size times p(1 − p), where p is its treated share, scaled to add up to 100%.

True SATE (simulation only)
Average of the true segment effects with the weights of D (simulation only)

What to notice

At the default settings, the individual effects have a standard deviation of 20 dollars. The orange curve with the Neyman standard error is wider than the bars. The average Neyman standard error is 3.61, and the true standard deviation is 2.92. So 98.1% of the intervals contain the SATE. Set the spread to 0. The curves now match the bars, and the share falls to about 95%.
Set the spread back to 20, and in the menu choose the target PATE. The bars widen until they match the orange curve, and about 95% of the Neyman intervals contain the PATE. In the table, the true variance now equals the average of the Neyman variance estimate.
Choose the target SATE again and set the share who get the offer to 20%. The green curve with the classical standard error becomes narrower than the bars, and only about 82% of the classical intervals contain the SATE. The Neyman intervals still contain it about 96% of the time. At a share of 50%, the two standard errors are equal.
In Part 2, the segment estimates weighted by segment size (C) have a standard deviation of 1.46, compared with 2.71 under complete randomization (A). Move the gap to 0: the two designs now spread about the same (1.46 and 1.43). Move it to 30: complete randomization spreads to 4.91, while blocking stays at 1.46.
Set the gap back to 15 and the rise in the treated share to 40%. The difference in means that ignores the segments (B) now averages 21.45, while the SATE is 10.00, because the treated group holds more loyal customers. Estimator C still averages 10.04. Then set the effect difference to 10. The fixed effects regression (D) falls to 6.54. The weights table shows why: D gives new and occasional customers, who have the smallest effects, more weight than their share.

The idea

A shop sends a discount offer to a random half of 100 customers. A computer draws which half gets the offer, and this page calls this draw the coin flips. We call the customers who get the offer the treated group and the others the comparison group. The outcome is the amount that each customer spends in the next month. The difference between the mean spending of the two groups estimates the effect of the offer.

Two targets are common. The sample average treatment effect (SATE) is the average effect among the 100 customers in the experiment. The population average treatment effect (PATE) is the average effect in the larger population that the customers come from (Imbens and Rubin 2015, ch. 6).

Each customer has two potential outcomes: \(Y(1)\), the spending with the offer, and \(Y(0)\), the spending without it. Neyman looked at the SATE in this way: keep the 100 customers and both of their potential outcomes fixed. Then let only the coin flips change. Over all possible coin flips, the difference in means averages to the SATE, so it is unbiased (Splawa-Neyman 1990; Imbens and Rubin 2015, ch. 6). Neyman derived its variance over the coin flips:

\[\frac{S_1^2}{n_1} + \frac{S_0^2}{n_0} - \frac{S_\tau^2}{n}\]

Here \(n_1\) customers get the offer, \(n_0\) do not, and \(n = n_1 + n_0\). \(S_1^2\) and \(S_0^2\) are the variances of \(Y(1)\) and \(Y(0)\) among the n customers. \(S_\tau^2\) is the variance of their individual effects. The data show each customer under one condition only. So the data can estimate \(S_1^2\) from the treated group and \(S_0^2\) from the comparison group. But they cannot estimate \(S_\tau^2\).

The Neyman standard error is the square root of \(s_1^2/n_1 + s_0^2/n_0\). Here \(s_1^2\) and \(s_0^2\) are the sample variances of the outcome in the two groups. It leaves out the last term of the formula. So its square, the variance estimate, is too large on average unless every customer has the same effect. In large samples, a 95% confidence interval built with it then contains the SATE in more than 95% of the experiments. Such an interval is called conservative.

The difference in means and its variance in symbols

\(D_i\) is 1 if customer \(i\) gets the offer. The difference in means is

\[\hat\tau = \frac{1}{n_1} \sum D_i Y_i - \frac{1}{n_0} \sum (1 - D_i) Y_i\]

With the \(n\) customers held fixed, \(E_D[\hat\tau] = \text{SATE}\), where \(E_D\) averages over the coin flips only. The variance over the coin flips is

\[\begin{aligned} V_D(\hat\tau) &= \frac{S_1^2}{n_1} + \frac{S_0^2}{n_0} - \frac{S_\tau^2}{n}, \\ \text{while} \quad E_D\Big[\frac{s_1^2}{n_1} + \frac{s_0^2}{n_0}\Big] &= \frac{S_1^2}{n_1} + \frac{S_0^2}{n_0} \end{aligned}\]

Now suppose that the customers are a random sample from a large population and that the target is the PATE. The random draw of the customers then adds variance. So the same standard error is correct on average (Imbens and Rubin 2015, ch. 6).

Standard errors from a regression

A regression of the outcome on an intercept and the treatment indicator gives the same estimate. Its standard error depends on the option that the software uses. The classical standard error assumes that the outcome has the same variance in both groups. It is wrong when the groups differ both in size and in variance. A robust standard error allows the variance of the outcome to differ between the groups. One version, HC2 (heteroskedasticity-consistent standard error of type 2), equals the Neyman standard error exactly (Ding 2024, ch. 4).

Blocking sorts the customers into groups of similar customers before the coin flips, for example new, occasional, regular, and loyal customers. These groups are the blocks, or segments. The coin flips then happen inside each segment. This design is a block randomized experiment, also called a stratified randomized experiment. The estimate is the difference in means inside each segment, averaged with weights equal to the segment shares of the customers. Its variance is the sum of the segment variances, each multiplied by the squared weight (Imbens and Rubin 2015, ch. 9).

In symbols, with blocks \(j = 1, \ldots, J\) of sizes \(n_j\) and segment estimates \(\hat\tau_j\), the block estimator is

\[\begin{aligned} & \hat\tau_B = \sum \frac{n_j}{n} \hat\tau_j, \\ & \text{with variance } \sum \Big(\frac{n_j}{n}\Big)^2 V(\hat\tau_j) \end{aligned}\]

Blocking removes the differences between the segments from the variance of the estimate. If the segments differ in average spending by large amounts and the customers inside a segment are similar, the estimate becomes more precise. If the segments do not predict spending, blocking gives no gain, and in a small sample it can lose a little precision (Ding 2024, ch. 5).

Different treated shares across segments and the fixed effects regression

Some designs give the offer to different shares of customers in different segments, for example to more of the loyal customers. Then the difference in means that ignores the segments is biased. The treated group holds more loyal customers, and loyal customers spend more even without the offer. The weighted average of the segment estimates stays unbiased.

A regression with a separate intercept for each segment, called segment fixed effects, also removes the differences between segments. This regression weights the segment estimates by \(n_j p_j (1 - p_j)\). Here \(n_j\) is the size of segment j, and \(p_j\) is its treated share (Angrist 1998). In symbols, the fixed effects coefficient equals

\[\frac{\sum \omega_j \hat\tau_j}{\sum \omega_j} \quad \text{with} \quad \omega_j = n_j p_j (1 - p_j)\]

These weights favor segments with a treated share near one half. Suppose that the effect differs across segments and that these weights differ from the segment shares. Then the regression estimates a weighted average of the segment effects that differs from the SATE.

References

Angrist, J. D. (1998). Estimating the labor market impact of voluntary military service using Social Security data on military applicants. Econometrica, 66(2), 249-288. https://doi.org/10.2307/2998558
Ding, P. (2024). A first course in causal inference. Chapman & Hall/CRC. Chapter 4, Neymanian Repeated Sampling Inference in Completely Randomized Experiments, and chapter 5, Stratification and Post-Stratification in Randomized Experiments. https://doi.org/10.1201/9781003484080. Free author version: arXiv:2305.18793.
Imbens, G. W., & Rubin, D. B. (2015). Causal inference for statistics, social, and biomedical sciences: An introduction. Cambridge University Press. Chapter 6, Neyman's Repeated Sampling Approach to Completely Randomized Experiments, and chapter 9, Stratified Randomized Experiments. https://doi.org/10.1017/CBO9781139025751
Splawa-Neyman, J. (1990). On the application of probability theory to agricultural experiments. Essay on principles. Section 9 (D. M. Dabrowska & T. P. Speed, Trans.). Statistical Science, 5(4), 465-472. https://doi.org/10.1214/ss/1177012031 (Original work published 1923)