03 · What You Need to Know
How to Evaluate Whether a Regression Discontinuity Cutoff Is Credible
The cutoff must belong to the actual assignment mechanism
The first requirement is that the threshold genuinely influences who receives the intervention.
For example, a university may use a diagnostic score of 60 to determine eligibility for additional tutoring. If the policy is consistently implemented, students immediately below and above the threshold face different treatment conditions or probabilities.
By contrast, a researcher cannot create a regression discontinuity design merely by dividing students into groups at the median diagnostic score after data collection.
Unless that division influenced treatment assignment, the threshold provides no treatment discontinuity to identify.
Researchers should obtain the original eligibility policy, administrative rules, and implementation records to establish how the cutoff operated.
The distinction is essential to understanding when regression discontinuity is appropriate.
The running variable must be clearly defined
The running variable is the measured quantity used to determine assignment relative to the cutoff.
It might be a diagnostic assessment score, age, income, institutional performance indicator, or another numerical measure.
Researchers should establish how the variable is measured, when it is recorded, whether its scale is consistent, and whether the recorded value is the same quantity used in the assignment decision.
For example, a university may use a standardized diagnostic score to determine tutoring eligibility, while the research dataset contains a later revised score.
If the revised score differs from the original assignment variable, analyzing the wrong measure can distort the estimated discontinuity.
The assignment variable should be reconstructed from reliable records rather than inferred from treatment receipt alone.
Why the cutoff should be established independently of observed outcomes
A credible assignment cutoff ordinarily exists as part of an intervention or policy rule before the outcomes being evaluated are observed.
For example, a diagnostic threshold may be established through an institutional support policy before students receive tutoring.
If researchers instead search through many possible thresholds and choose the one producing the largest achievement difference, the resulting discontinuity does not have the same causal interpretation.
Outcome-driven threshold selection can introduce substantial analytical bias.
Researchers should distinguish genuine administrative cutoffs from exploratory thresholds introduced solely for analysis.
Where a policy cutoff changes over time, the timing and reasons for those changes should be documented.
The cutoff must create a treatment discontinuity
Regression discontinuity requires a change in treatment assignment or treatment probability at the threshold.
In a sharp design, treatment status follows the cutoff rule deterministically.
In a fuzzy design, treatment probability changes discontinuously, but some units do not comply with the eligibility rule or receive treatment through exceptions.
| Assignment situation |
Implication |
Researcher action |
| All eligible students receive treatment and all ineligible students do not |
Potential sharp regression discontinuity |
Verify actual assignment and continuity assumptions. |
| Eligibility changes treatment probability but compliance is imperfect |
Potential fuzzy regression discontinuity |
Estimate the treatment probability discontinuity and evaluate additional assumptions. |
| Eligibility threshold exists but treatment rates do not change |
Little or no identifying discontinuity |
Reconsider whether regression discontinuity is feasible. |
| Several interventions begin at the same cutoff |
Separate effects may be unidentified |
Determine whether the estimand concerns a combined intervention package. |
| Cutoff is invented after observing outcomes |
No genuine threshold-based assignment mechanism |
Do not describe the arbitrary division as regression discontinuity. |
The observed treatment probability should be examined on both sides of the cutoff. The official rule alone is not sufficient evidence that implementation followed it.
Why precise manipulation threatens the design
One of the most consequential threats occurs when units can precisely influence whether they fall above or below the threshold.
Suppose instructors know that students scoring below 60 receive additional support and can adjust borderline diagnostic scores.
Students assigned just below the cutoff may then differ systematically from those just above it because of instructor discretion, student requests, or other factors.
If these factors also influence outcomes, the continuity-based causal argument may fail.
Researchers should investigate whether the running variable was recorded before eligibility decisions, whether scoring was standardized, and whether appeals or reassessments were possible.
However, not every form of sorting automatically invalidates regression discontinuity. The relevant issue is whether sorting undermines continuity of the potential outcome functions at the threshold.
How density tests help assess sorting
Researchers may examine whether the distribution of the running variable changes unexpectedly at the cutoff.
For example, an unusually large number of students scoring 59 and very few scoring 60 could suggest strategic scoring or eligibility manipulation.
Density discontinuity tests, including procedures associated with McCrary's work and subsequent developments, can help identify suspicious patterns.
However, these tests require appropriate treatment of the running variable's distribution, discreteness, and measurement properties.
A density discontinuity is not definitive proof of invalidity, and a nonsignificant test does not establish that manipulation is absent.
The statistical evidence should be interpreted alongside administrative knowledge of the scoring and assignment process.
Why predetermined characteristics should be examined
Under a credible continuity-based design, relevant pretreatment characteristics should generally change smoothly through the cutoff.
Researchers may examine prior achievement, socioeconomic indicators, demographic characteristics, or other variables measured before treatment.
For example, if students immediately below the tutoring threshold have substantially different prior achievement histories from students immediately above it, the discontinuity may warrant investigation.
Such differences could indicate sorting, another assignment rule, or an unusual feature of the population.
However, balance tests are diagnostics rather than definitive validation procedures.
Failure to detect a difference does not prove that all relevant potential outcomes are continuous.
Other interventions at the same cutoff can prevent separate attribution
Suppose students scoring below 60 receive AI-supported tutoring, additional instructor consultations, and financial assistance.
All three interventions begin at the same diagnostic threshold.
If subsequent achievement changes discontinuously at 60, the observed jump may reflect the combined effects of these services.
Without additional identifying variation or assumptions, researchers generally cannot attribute the entire discontinuity specifically to AI tutoring.
They may instead define the treatment as eligibility for the combined support package.
Researchers should examine all policies and services linked to the threshold before specifying the causal estimand.
Why the cutoff should not coincide with unrelated outcome discontinuities
A suitable cutoff requires a credible argument that relevant untreated potential outcomes are continuous at the threshold.
For example, a diagnostic score of 60 may also determine academic probation status, scholarship eligibility, or mandatory course placement.
These policies could independently influence later achievement.
If they change at exactly the same threshold as tutoring, the observed outcome discontinuity may not isolate tutoring's effect.
Researchers should therefore investigate the broader institutional consequences of crossing the cutoff.
The numerical threshold may be clear while the causal interpretation remains ambiguous.
Why enough observations must exist near the threshold
Regression discontinuity estimation relies heavily on observations near the cutoff.
A dataset may contain thousands of students overall but relatively few students with scores close to 60.
For example, if most students score below 30 or above 85, the available information near the cutoff may be limited.
Researchers should examine the distribution of the running variable and conduct design-specific power or precision calculations.
Large total sample size does not guarantee precise estimation of a local discontinuity.
Why discrete running variables deserve special attention
Many educational assignment variables are discrete rather than continuously measured.
Diagnostic scores may take integer values, grade categories, or a limited set of possible totals.
If only a few distinct score values exist near the cutoff, conventional local polynomial approximations may depend heavily on functional form and extrapolation.
For example, a score scale containing only five categories may provide insufficient support for standard continuity-based regression discontinuity methods.
Researchers should report the number of distinct running-variable values near the threshold and consider methods appropriate to discrete support.
The existence of observations on both sides does not automatically establish that conventional asymptotic approximations are reliable.
Why the cutoff must be interpreted in relation to treatment timing
The running variable should be measured at a time appropriate to the assignment mechanism.
For example, a diagnostic assessment used to determine tutoring eligibility should ordinarily precede tutoring exposure.
If the score is measured after students have already received intervention components, it may itself be affected by treatment.
Using a post-treatment score as though it were an unaffected assignment variable can compromise the causal design.
Researchers should establish the chronological sequence of assessment, eligibility determination, intervention delivery, and outcome measurement.
How cutoff exceptions affect interpretation
Some institutions allow administrators to override eligibility rules.
For example, students scoring 60 or above may receive tutoring after an instructor recommendation, while some students below 60 may decline participation.
Such exceptions do not necessarily invalidate the design, but they may change it from sharp to fuzzy regression discontinuity.
The researcher must establish whether the probability of treatment still changes discontinuously at the cutoff.
For a conventional fuzzy design, interpreting the ratio of outcome and treatment discontinuities as a local treatment effect requires additional assumptions.
Researchers should avoid treating eligibility and actual treatment receipt as interchangeable.
Why cutoff validity is not established by the outcome graph
A graph showing a visible jump in achievement at the threshold may appear persuasive.
However, graphical discontinuities can arise from sampling variability, smoothing choices, other policies, or inappropriate functional forms.
Conversely, a genuine treatment effect may not produce a visually obvious jump in a noisy dataset.
Researchers should evaluate the assignment mechanism independently of the observed outcome pattern.
Graphical presentation is useful for transparency but cannot replace identification assumptions and appropriate statistical inference.
What if several cutoffs exist?
Some interventions use different thresholds across institutions, years, or populations.
For example, universities may adopt different diagnostic score cutoffs because their tutoring capacities differ.
Multiple-cutoff regression discontinuity designs may be possible, but they require careful definition of the running variable, cutoff-specific comparisons, and aggregation of local effects.
Researchers should not automatically combine all observations into one analysis without accounting for the assignment rules.
Differences between cutoff populations may also limit the interpretation of a pooled estimate.
When a cutoff is unsuitable despite being formally documented
A threshold may be unsuitable when it does not meaningfully affect treatment, when the assignment variable is extensively manipulated, or when the outcome discontinuity cannot be separated from other simultaneous policy changes.
It may also provide inadequate information when very few observations occur near the cutoff or when the running variable has insufficient distinct values for the intended estimator.
Researchers should assess these issues before treating regression discontinuity as the preferred design.
If the cutoff does not support credible local identification, another quasi-experimental strategy may be more appropriate.
Watch Out
A documented eligibility threshold is not proof of regression discontinuity validity. Researchers must establish that the cutoff creates a genuine treatment discontinuity and that other mechanisms do not undermine the local causal comparison.
