1.1 Randomized experiments
Randomization inference: an exact test for a small experiment
A small experiment has too few units for the usual large-sample formulas, so randomization inference uses only the way the treatment was assigned. This page shows how the test works and how the choice of test statistic changes the answer. It also shows how a series of tests gives a confidence interval.
In brief
An online shop tests a new product page with 10 customers. A computer picks 5 of them at random to see the new page, and the other 5 see the old page. The test asks whether the new page changes the spending of any customer in the next month.
- A test statistic is one number computed from the data, for example the difference in means between the two groups.
- The randomization distribution is the set of values of the statistic for all 252 possible assignments. Every outcome stays fixed, as the sharp null hypothesis of no effect for any customer says.
- The p-value is the share of these values that are at least as extreme as the value for the actual assignment. The test rejects the sharp null at the 5% level if the p-value is 0.05 or less.
- The 95% confidence interval from inverting the test is the set of constant effects that the test does not reject at the 5% level. A constant effect is the same for every customer.
Move the sliders and watch these numbers. The explanation below the simulation gives the reasons.
Try it
The simulation has 10 customers at first. The sliders set the true effect of the new page, the extra spending of a large customer, and the number of customers. Two menus set the test statistic and the kind of test.
How the simulation makes the data
Without the new page, the spending of each customer is drawn at random around 40 dollars. Customer 1 is a large customer: the large-customer slider adds an amount to the spending of this customer with either page. The new page adds the true effect to the spending of every customer. The simulation keeps both potential outcomes of every customer, so it knows the true effect.
The data
The randomization distribution under the sharp null of no effect
Inverting the test: one p-value for each constant effect
| Test statistic | Value for the actual assignment | p-value | Rejections at 5% if the sharp null were true |
|---|---|---|---|
| Difference in means (dollars) | |||
| Difference in medians (dollars) | |||
| Difference in mean ranks |
The bold row is the chosen statistic, and the p-values use the chosen kind of test. A real study chooses one statistic before it sees the outcomes. The table shows all three only to compare them. The last column assumes that the new page changes the spending of no customer. Then each customer would spend the observed amount under any assignment. The column shows the share of the assignments that the test uses in which the test would then give a p-value at or below 0.05.
- Possible assignments
- Assignments that the test uses
- Assignments at least as extreme as the actual one
- p-value
- Smallest possible p-value
- 95% confidence interval from inverting the test (dollars)
- Middle of the effects with the highest p-value (dollars)
- True effect (dollars)
The confidence interval always uses the two-sided test of the chosen statistic. Its ends are values of τ0, so they are in dollars for every statistic. The p-value, the 95% confidence interval, and other terms are defined in the key terms on the home page.
The idea
An online shop tests a new product page with 10 customers. A computer picks 5 of the 10 at random to see the new page, and the other 5 see the old page. We call the first set of customers the treated group and the second set the comparison group. The outcome is the spending of each customer in the next month.
Randomization inference needs no large-sample approximation. It uses only the rule that assigned the page (Imbens and Rubin 2015, ch. 5).
A null hypothesis is the claim that a test checks against the data. This test starts from the null hypothesis of Fisher that the treatment has no effect for any customer. Write \(Y_i(1)\) for the spending of customer i with the new page and \(Y_i(0)\) for the spending with the old page. These are the potential outcomes of customer i.
The null hypothesis says that \(Y_i(1) = Y_i(0)\) for every customer i. It is called the sharp null hypothesis, because it fixes both potential outcomes of every customer (Imbens and Rubin 2015, ch. 5). Then each customer would spend the same amount with either page, so we know what the data would show under any other assignment.
The sharp null says more than a zero average effect. Suppose that the new page raises the spending of one customer by 5 dollars and lowers that of another by 5 dollars. Then the average effect is zero, but the sharp null is false. So a rejection of the sharp null is evidence that the new page changes the spending of at least one customer. The test is not built to test a zero average effect (Ding 2024, sec. 8.1).
The test has four steps (Imbens and Rubin 2015, ch. 5):
- Choose a test statistic before looking at the outcomes. A test statistic is one number computed from the data. An example is the mean spending of the treated group minus the mean spending of the comparison group.
- Compute the statistic for the actual assignment.
- Compute it again for every other assignment that the same rule could have produced. Keep the outcome of every customer fixed, as the sharp null says. With 10 customers and 5 treated, there are 252 assignments. Their 252 values form the randomization distribution.
- Compute the p-value: the share of the 252 values that are at least as extreme as the value for the actual assignment.
This p-value is exact. The test rejects the null hypothesis at the 5% level if the p-value is 0.05 or less. A two-sided test compares absolute values, so large values of either sign count.
Notation for the test
\(D_i\) is 1 if customer \(i\) sees the new page and 0 otherwise. The design gives the new page to \(n_1\) of \(n\) customers. All \(K = C(n, n_1)\) assignments \(d_1, \ldots, d_K\) are equally likely. \(T(d, y)\) is the test statistic for assignment \(d\) and outcomes \(y\). Under the sharp null of no effect, the observed outcomes \(y\) stay the same under every assignment.
In symbols, the one-sided p-value is
\[p = \frac{1}{K} \times (\text{number of } k \text{ with } T(d_k, y) \ge T(D, y))\]
The two-sided p-value uses \(|T|\) in place of \(T\).
The smallest possible p-value
The actual assignment is one of the 252, so the p-value is never below 1/252. When both groups have the same size, every assignment has a mirror image in which the two groups change places. The mirror image has the same absolute value, so the smallest two-sided p-value is at least 2/252. For the difference in means it is exactly 2/252. With 6 customers, it is at least 2/20, or 0.10, so the test can never reject at the 5% level.
If the researcher chooses the test statistic before seeing the outcomes, the test is valid. If the sharp null is true, the test then rejects it at the 5% level in at most 5% of the assignments (Ding 2024, ch. 3). A researcher who tries several statistics and reports the smallest p-value does not have a valid test (Imbens and Rubin 2015, ch. 5).
The choice of statistic still matters, because it changes the power of the test. The power is the probability that the test rejects the null when the null is false. A few large values can move the difference in means by a large amount. They move the difference in medians and the difference in mean ranks less. Ranks order the customers by spending, from 1 for the lowest to 10 for the highest (Imbens and Rubin 2015, ch. 5; Cunningham 2021, ch. 4). No statistic has the most power in every situation.
The same steps also test the sharp null of a constant effect. This null says that \(Y_i(1) = Y_i(0) + \tau_0\) for every customer, for a chosen number \(\tau_0\). Subtract \(\tau_0\) from the outcome of each treated customer. Under this null, the result is \(Y_i(0)\) for every customer, so the test of no effect applies.
In symbols, the test of the sharp null \(Y_i(1) = Y_i(0) + \tau_0\) uses the adjusted outcomes \(y_i - D_i \tau_0\) in place of \(y\).
Repeat the test for many values of \(\tau_0\). The values that the test does not reject at the 5% level form a 95% confidence interval. This method is called inverting the test (Imbens and Rubin 2015, ch. 5).
In symbols, the 95% confidence interval is the set of all \(\tau_0\) with a two-sided p-value above 0.05.
The interval assumes that every customer has the same effect. If the effects differ across customers, the interval need not contain the average effect 95% of the time (Ding 2024, ch. 3 and sec. 8.1).
Module 5.3 uses the same logic for dates: it ranks the estimate on the policy date among the estimates on dates when nothing happened.
Larger experiments: random draws of assignments
With 100 customers and 50 treated, the number of assignments has 30 digits. Then a computer draws many assignments at random, for example 10,000. The p-value is the share of these draws that are at least as extreme as the actual assignment (Imbens and Rubin 2015, ch. 5). If the actual assignment counts as one of the draws, the p-value is never 0, and the test stays valid (Ding 2024, ch. 3).
References
- Cunningham, S. (2021). Causal inference: The mixtape. Yale University Press. Chapter 4, Potential Outcomes Causal Model. https://doi.org/10.2307/j.ctv1c29t27. Free online first edition: chapter 4, section 4.2.
- Ding, P. (2024). A first course in causal inference. Chapman & Hall/CRC. Chapter 3, The Completely Randomized Experiment and the Fisher Randomization Test, and section 8.1, Testing Strong and Weak Null Hypotheses in the CRE. https://doi.org/10.1201/9781003484080. Free author version: arXiv:2305.18793.
- Imbens, G. W., & Rubin, D. B. (2015). Causal inference for statistics, social, and biomedical sciences: An introduction. Cambridge University Press. Chapter 5, Fisher's Exact P-Values for Completely Randomized Experiments. https://doi.org/10.1017/CBO9781139025751