4.1 Regression discontinuity
Regression discontinuity: estimating a jump at a cutoff
Regression discontinuity (RD) estimates the effect of a treatment when a score decides who gets it. Units with a score at or above a cutoff get the treatment, and units below the cutoff do not. This page shows how two things change the estimate: the curves that the method fits, and units that move their score across the cutoff.
In brief
A university gives a scholarship to every student with an entrance test score of 50 or more. We want to know how the scholarship changes the first-year grade average of a student.
- On each side of the cutoff at 50, the method fits a curve and reads its value there. The RD estimate is the right value, on the scholarship side, minus the left value. It estimates the jump in the average grade at the cutoff.
- The true effect at the cutoff is the amount that the scholarship adds to the grade of every student at or above the cutoff. The simulation knows it.
- Estimate minus true effect shows the error of the estimate in the current sample. This error is the bias plus the sampling error, the part of the error that changes from sample to sample. The bias is the average of the error over many samples.
- The density check p-value tests whether as many students have a score from 48 to under 50 as from 50 to under 52. Students who raise their score make this split uneven.
Move the sliders and watch these numbers. The explanation below the simulation gives the reasons.
Try it
The simulation draws students with test scores spread evenly from 0 to 100. The sliders set the true effect, a smooth bend in the average grade, the bandwidth, and the share of students who raise their score. The Estimator list chooses how the method fits the curves.
How the simulation makes the data
Each student also has an effort level that the researcher does not observe. Effort raises the grade. Effort is not related to the score, unless you let some students raise their score.
The average grade rises with the score, and the curvature slider adds a smooth bend around the score of 50. The bend has no jump, so it is not an effect of the scholarship. The scholarship adds the true effect to the grade of every student with a score of 50 or more.
In symbols, write \(X\) for the score and \(Y\) for the grade. Let \(D = 1\) for a student with the scholarship and \(D = 0\) otherwise. The simulation draws \(X\) from the uniform distribution on \([0, 100]\) and sets
\[\begin{aligned} Y = {} & 60 + 0.2(X - 50) \\ & + 6k \tanh((X - 50)/8) \\ & + 2e + \tau D + \sigma\varepsilon \end{aligned}\]
Here \(\tanh\) is the hyperbolic tangent, \(k\) is the curvature, \(\tau\) is the true effect, and \(\sigma\) is the noise. The effort \(e\) and the noise term \(\varepsilon\) are independent standard normal draws. With manipulation, each student with \(45 \le X < 50\) and \(e > 0\) moves with probability \(q\). The new score comes from the uniform distribution on \([50, 52)\). The simulation then uses the new score as \(X\) in the formula above.
The simulation knows the true effect. With the default settings, the bias of the local linear estimate is about 0.6, because the straight lines cannot follow the bend. Draw new samples to see the sampling error change. The standard error in the table measures how much an estimate changes from sample to sample.
The true curve in this simulation is smooth, and a polynomial of order 4 can follow it closely. So here the global polynomial has little bias. Watch instead how much its estimate changes from sample to sample.
- Students with a score from 48 to under 50
- Students with a score from 50 to under 52
- Density check p-value
| Side of the cutoff | Fitted average grade at 50 | Standard error | Students used |
|---|---|---|---|
| Below 50 (no scholarship) | |||
| 50 or above (scholarship) | |||
| RD estimate (right minus left) |
- RD estimate
- 95% confidence interval
- True effect at the cutoff
- Estimate minus true effect
- Students used in the fit
The idea
A university gives a scholarship to every student with an entrance test score of 50 or more. Scores go from 0 to 100. We want to know how the scholarship changes the first-year grade average of a student.
A comparison of all students with the scholarship and all students without it mixes two things. One thing is the effect of the scholarship. The other thing is the difference between students with high and low test scores. These students would get different grades even without a scholarship.
The test score is the running variable: the variable whose value decides who gets the treatment. The value 50 is the cutoff. The design is sharp: every student with a score at or above the cutoff gets the scholarship. No student below the cutoff gets it. So a student with a score of exactly 50 gets the scholarship (Cunningham 2021, ch. 6; Cattaneo, Idrobo, and Titiunik 2020).
Students with a score just below 50 and students with a score just above 50 have almost the same score. RD compares the average grade on the two sides of the cutoff, at the cutoff itself. It estimates the jump in the average grade at the cutoff.
The jump measures the effect of the scholarship if the continuity assumption holds. Think of the average grade that students with a given score would get without the scholarship. This average must change smoothly with the score at 50, with no jump. The same must be true for the average grade with the scholarship. Under this assumption, the jump at the cutoff equals the average effect of the scholarship for students who score 50 (Cattaneo, Idrobo, and Titiunik 2020).
In symbols, write \(X\) for the score and \(c = 50\) for the cutoff. Let \(D = 1\) if \(X \ge c\) and \(D = 0\) if \(X < c\). Write \(Y(1)\) for the grade of a student with the scholarship and \(Y(0)\) for the grade of the same student without it. The observed grade is \(Y = D\,Y(1) + (1 - D)\,Y(0)\). The target is the average effect at the cutoff:
\[\tau = E[Y(1) - Y(0) \mid X = c]\]
If \(E[Y(0) \mid X = x]\) and \(E[Y(1) \mid X = x]\) are continuous at \(x = c\), then
\[\tau = \lim_{x \downarrow c} E[Y \mid X = x] - \lim_{x \uparrow c} E[Y \mid X = x]\]
Continuity also fails when something else changes at the same cutoff. Suppose that the university also puts every student with a score of 50 or more into an honors program. Then the jump at 50 mixes the effect of the scholarship with the effect of the program. The data cannot separate the two (Cunningham 2021, ch. 6).
The result is local. It is the average effect for students at the cutoff. Without more assumptions, it says nothing about students far from the cutoff, for example students with a score of 90.
Students with the scholarship and students without it never have the same score. So RD cannot compare the two groups at one score value. Instead, RD fits a curve to the data on each side and reads the value of each curve at the cutoff. The cutoff is at the edge of the data on each side. So this step extrapolates each curve to the cutoff from one side (Cattaneo, Idrobo, and Titiunik 2020).
How to fit the curves
The default method is local linear regression (Cattaneo, Idrobo, and Titiunik 2020). It uses only the students with a score within a distance h of the cutoff. This distance is the bandwidth. The range from \(50 - h\) to \(50 + h\) is the bandwidth window. So h is the distance on each side of the cutoff, and the bandwidth window is \(2h\) wide.
On each side of the cutoff, the method fits a separate straight line by weighted least squares. This is a regression in which some students count more than others. A kernel is a rule that sets each weight from the distance between the score and the cutoff. The triangular kernel gives the most weight to students at the cutoff, and the weight falls in a straight line to zero at distance h. Cattaneo, Idrobo, and Titiunik (2020) recommend this kernel, and they note that results usually change little with the choice of kernel. The RD estimate is the value of the right line at 50 minus the value of the left line at 50.
In symbols, the triangular kernel is \(K(u) = 1 - |u|\) for \(|u| \le 1\) and \(K(u) = 0\) otherwise. On each side of the cutoff, local linear regression with bandwidth \(h\) chooses \(a\) and \(b\) to minimize
\[\sum_i K\!\left(\frac{X_i - c}{h}\right) \bigl(Y_i - a - b(X_i - c)\bigr)^2\]
over the students on that side. The RD estimate is \(a_{\text{right}} - a_{\text{left}}\).
The RD estimate from one regression
The same estimate comes from one weighted regression on all students in the bandwidth window, with the same kernel weights. The regression is \(Y = \alpha + \beta(X - c) + \tau D + \gamma(X - c)D\). Because the score is centered at the cutoff, the coefficient \(\tau\) on \(D\) is the RD estimate (Cunningham 2021, ch. 6).
The bias of a method is the average of its estimates over many samples minus the true effect. The bandwidth sets a trade-off. A smaller bandwidth uses students closer to the cutoff, where a straight line follows the true curve better, so the bias falls.
A smaller bandwidth also uses fewer students, so the estimate changes more from sample to sample. Then the 95% confidence interval, a range built to contain the true effect in about 95% of samples, gets wider. Because of this trade-off, researchers should choose the bandwidth with a data-driven rule (Cattaneo, Idrobo, and Titiunik 2020). On this page you move the bandwidth by hand to see the trade-off.
Several bandwidths and the confidence interval
Researchers often report the estimate for several bandwidths. The bandwidth chart on this page does this for the current sample. As the bandwidth grows, the interval gets shorter. With a curved outcome line, the estimate also moves away from the true effect, because the bias grows. This check is useful mainly for bandwidths close to a data-driven choice (Cattaneo, Idrobo, and Titiunik 2020).
The usual 95% confidence interval measures only how much the estimate changes from sample to sample. It does not include the bias. When the bias is large compared with the standard error, the interval misses the true effect far more often than in 5% of samples. At the bandwidth that minimizes the mean squared error, the bias stays about as large as the standard error, even in large samples. So Cattaneo, Idrobo, and Titiunik (2020) recommend robust bias-corrected confidence intervals for inference at that bandwidth. This page shows only the usual interval.
Each side of the cutoff has a heteroskedasticity-robust standard error (se), which allows the noise to differ across students. The page combines the two as \(\text{se} = \sqrt{\text{se}_{\text{left}}^2 + \text{se}_{\text{right}}^2}\). The 95% confidence interval is the estimate \(\pm 1.96\,\text{se}\).
Other ways to fit the curves
A simpler method compares the average grades on the two sides of the cutoff inside the bandwidth window. This method is local constant regression. At the cutoff, the bias of local constant regression is roughly proportional to the bandwidth when the bandwidth is small. The reason is that the grade changes with the score inside the bandwidth window. This behavior at the edge of the data is a reason why local linear regression is the default (Cattaneo, Idrobo, and Titiunik 2020). Choose Local constant in the simulation to see this bias.
In symbols, local constant regression drops the slope terms.
Local quadratic regression fits a curve with a squared term, \((\text{score} - 50)^2\), on each side instead of a straight line. It can follow a bend better, but its estimate changes more from sample to sample (Cattaneo, Idrobo, and Titiunik 2020). In symbols, local quadratic regression adds a term in \((X_i - c)^2\).
Another method fits one global polynomial to all the data on each side. An example is a polynomial of order 4, which has terms up to the fourth power of the score. In symbols, the global polynomial uses terms up to \((X_i - c)^4\), all students on each side, and equal weights. Gelman and Imbens (2019) give three reasons against this method.
First, the estimate is a weighted average of the grades at or above the cutoff minus a weighted average of the grades below it. The weights depend only on the scores and the cutoff, not on the grades. With a global polynomial of high order, these weights can have unattractive properties. For example, students far from the cutoff can have a large influence on the estimate (Cattaneo, Idrobo, and Titiunik 2020).
Second, the estimate can change when the order of the polynomial changes. Third, its confidence intervals can be misleading. Gelman and Imbens recommend local linear or local quadratic fits instead.
When students can move their score
Suppose that some students who would score just below 50 can raise their score to just above 50. This is manipulation of the running variable. These students can differ from other students, for example because they work harder. Then the students just above the cutoff are no longer like the students just below it, and the continuity assumption can fail (McCrary 2008).
For example, students who score just below 50 may retake the test. Suppose that students with more time or money retake it more often. Then the students just above the cutoff differ from the students just below it (Cunningham 2021, ch. 6).
The density of the score tells how common each score is. A histogram of the score shows the density. This kind of manipulation leaves a trace in the histogram: too few students just below the cutoff and too many just above it. McCrary (2008) proposed a formal test for a jump in the density of the running variable at the cutoff.
The density check and its limits
This page uses a simpler check that Cattaneo, Idrobo, and Titiunik (2020) describe. It counts the students with a score from 48 to under 50 and the students with a score from 50 to under 52. Without manipulation, a student with a score from 48 to under 52 is about equally likely to be on either side of the cutoff.
An exact binomial test gives the p-value. The p-value is the probability of a split at least as uneven as the observed split. This probability assumes that each student has a chance of 1/2 to be on each side. In an example in their text, 47 observations below the cutoff and 53 above it give a p-value of 0.6173. This value gives no evidence of manipulation.
The density check tests a consequence of one condition: students cannot control their score precisely enough to choose their side of the cutoff. It does not test the continuity assumption itself. A smooth density does not prove that the design is valid, and a valid design does not need a smooth density (McCrary 2008). For example, if as many scores move down across the cutoff as move up, the histogram can stay smooth while the estimate is biased. If chance alone pushes some students across the cutoff, the histogram jumps, but the estimate need not be biased.
References
- Cattaneo, M. D., Idrobo, N., & Titiunik, R. (2020). A practical introduction to regression discontinuity designs: Foundations. Cambridge University Press. https://doi.org/10.1017/9781108684606
- Cunningham, S. (2021). Causal inference: The mixtape. Yale University Press. Chapter 6, Regression Discontinuity. https://doi.org/10.2307/j.ctv1c29t27. Free online first edition: chapter 6.
- Gelman, A., & Imbens, G. (2019). Why high-order polynomials should not be used in regression discontinuity designs. Journal of Business & Economic Statistics, 37(3), 447-456. https://doi.org/10.1080/07350015.2017.1366909
- McCrary, J. (2008). Manipulation of the running variable in the regression discontinuity design: A density test. Journal of Econometrics, 142(2), 698-714. https://doi.org/10.1016/j.jeconom.2007.05.005