03 · What You Need to Know
When Quasi-Experimental Designs Are Appropriate and What Makes Them Credible
What is a quasi-experimental design?
A quasi-experimental design is an approach to evaluating causal effects when treatment assignment is not controlled through random allocation by the researcher.
Rather than relying on randomization to establish comparability, quasi-experimental research uses features of the assignment process, comparison structure, or outcome history to estimate what would have happened without the intervention.
For example, a policy may become effective at a specified date, an educational program may be allocated using an eligibility threshold, or a government intervention may be introduced in some regions before others.
These circumstances can provide useful variation in intervention exposure, but their causal value depends on the design's identification assumptions.
Quasi-experimental research is therefore not simply experimental research without randomization. Its defining methodological challenge is to establish a credible counterfactual through another strategy.
Why the counterfactual matters
A causal effect concerns the difference between what would happen under an intervention and what would happen without it.
For an individual or institution, only one of these potential outcomes can ordinarily be observed under a given intervention decision at a particular time.
If a school adopts a digital learning platform, researchers observe its outcomes under adoption but cannot simultaneously observe what those same outcomes would have been without the platform.
The unobserved alternative is the counterfactual.
Randomized trials use chance assignment to support comparisons between groups. Quasi-experimental designs instead rely on assumptions about treatment assignment, outcome trajectories, thresholds, or other identifying features.
The central question is whether the proposed comparison approximates the relevant counterfactual sufficiently well for the intended causal conclusion.
When randomization is ethically or operationally infeasible
Some interventions cannot realistically be randomized because implementation decisions have already been made or because allocation is governed by policy, institutional requirements, or ethical constraints.
For example, a government may introduce a mandatory educational reform across selected regions. Researchers may have no authority to decide which schools receive the reform.
Similarly, a university may implement a student support policy according to administrative priorities that cannot be altered for research purposes.
Quasi-experimental evaluation may be appropriate when these circumstances create usable comparison information.
However, researchers should distinguish genuine constraints from convenience. Randomization should not be dismissed merely because administrative allocation is easier.
When random assignment remains feasible and ethically appropriate, a randomized design may offer stronger protection against confounding.
When an intervention has already been implemented
Quasi-experimental research is often useful for evaluating policies or programs introduced before researchers became involved.
For example, a university may have introduced an AI literacy requirement two academic years earlier. Administrative records might contain student outcomes from several years before and after implementation.
These data could support an interrupted time-series analysis if the intervention date is identifiable, the outcome is measured consistently, and sufficient observations are available to estimate pre-intervention patterns.
Alternatively, comparable universities that did not introduce the requirement might provide a control series.
Retrospective evaluation does not automatically weaken a design, but missing documentation, changes in measurement, and concurrent interventions may make causal identification more difficult.
The availability of historical data is useful only when the data can support the assumptions of a suitable design.
When a credible comparison group is available
Many quasi-experimental designs compare units receiving an intervention with units that do not receive it or receive it later.
However, untreated units are not necessarily appropriate comparison units.
Suppose universities adopting an AI advising system have stronger infrastructure and more experienced support staff than nonadopting universities. These differences may independently influence student retention.
Simply comparing retention rates could therefore confound intervention effects with institutional characteristics.
A credible comparison group should represent a plausible approximation of the treated group's outcomes in the absence of treatment, under the selected design's assumptions.
That may require comparable pre-intervention trajectories, similar eligibility rules, or a defensible assignment mechanism.
The question of what makes a comparison group credible is consequently central to quasi-experimental design selection.
Which quasi-experimental designs are commonly used?
Several approaches can support causal evaluation without randomized assignment. They rely on different sources of identifying variation and should not be treated as interchangeable.
| Design |
When it may be appropriate |
Central identification concern |
| Difference-in-differences |
Treated and comparison groups are observed before and after intervention. |
Credible counterfactual outcome trends, including parallel trends under the relevant assumptions. |
| Regression discontinuity |
Treatment eligibility or receipt changes at a known cutoff. |
Continuity of relevant potential outcomes at the threshold and absence of problematic manipulation. |
| Interrupted time series |
Repeated outcomes are available before and after a clearly timed intervention. |
Credible modeling of the counterfactual trend and competing events. |
| Controlled interrupted time series |
A suitable comparison series is available alongside repeated treated outcomes. |
Whether the control captures relevant background changes without intervention spillover. |
| Instrumental-variable design |
A variable changes treatment exposure through a defensible source of variation. |
Instrument relevance and exclusion, independence, and other assumptions appropriate to the estimand. |
| Comparative interrupted or staggered rollout evaluation |
Institutions adopt an intervention at different times. |
Selection into timing, anticipation, temporal trends, and heterogeneous treatment effects. |
Some methodological traditions also describe non-equivalent control-group designs and simple pretest-posttest arrangements as quasi-experimental. These labels describe allocation and measurement structures but do not by themselves establish strong causal identification.
A nonrandomized pretest-posttest comparison may be considerably weaker than a carefully implemented regression discontinuity or controlled interrupted time-series study.
When difference-in-differences may be appropriate
Difference-in-differences is useful when researchers observe outcomes for treated and comparison groups before and after an intervention and can defend assumptions about their untreated outcome trajectories.
For example, a university introduces a student support program while another university continues existing services. Researchers compare how retention changes across the two institutions.
If the comparison university provides a credible estimate of how retention would have changed at the treated university without the program, the difference between their changes may identify an intervention effect.
However, the approach is vulnerable when treated and comparison institutions would have followed different trends because of unrelated changes.
In staggered adoption settings, treatment effects may vary across implementation cohorts and exposure durations. Modern estimators may be needed rather than relying automatically on conventional two-way fixed-effects regression.
The conditions for using difference-in-differences should be evaluated before selecting the method.
When regression discontinuity may be appropriate
Regression discontinuity exploits an assignment rule based on a numerical threshold.
Suppose students scoring below 60 on a diagnostic assessment become eligible for additional AI-supported tutoring, while students scoring 60 or above do not.
Students immediately below and above the cutoff may be similar in important respects, except for their probability of receiving tutoring.
Under suitable continuity and assignment assumptions, the discontinuity in outcomes at the threshold can identify a local causal effect.
However, the estimate generally concerns units near the cutoff rather than all students.
Researchers must also investigate whether the assignment variable could be precisely manipulated and whether other policies change at the same threshold.
A cutoff is useful only when it supports the assumptions required for regression discontinuity.
When interrupted time series may be appropriate
Interrupted time-series analysis evaluates whether the level or trajectory of an outcome changes following an intervention introduced at an identifiable time.
For example, a university may have monthly academic advising completion rates for several years before and after introducing a new platform.
Researchers can model the pre-intervention trajectory and examine whether subsequent observations depart from the projected counterfactual.
The design becomes more credible when outcome measurement is consistent, the intervention timing is clear, and alternative explanations for changes at the intervention date are addressed.
Seasonality, autocorrelation, changes in measurement, and concurrent events require particular attention.
A suitable comparison series can strengthen the design when it captures background influences that would otherwise be confused with the intervention.
Why having a pretest and posttest is not enough
A common misconception is that measuring outcomes before and after an intervention automatically establishes a quasi-experimental design capable of supporting causal conclusions.
Suppose one class receives a digital learning intervention and its average achievement increases from 65 to 75.
The ten-point improvement could reflect the intervention, ordinary learning, assessment differences, maturation, or other instructional experiences.
Without a credible counterfactual, the observed change cannot be attributed confidently to treatment.
Even adding a comparison class may be insufficient if the classes differ substantially or would have followed different outcome trajectories.
The relevant issue is not the number of measurements alone, but whether the design addresses plausible alternative explanations.
Why selection into treatment is a central threat
In nonrandomized evaluations, treatment recipients often differ systematically from nonrecipients.
Students may volunteer for tutoring because they are more motivated. Schools may receive additional support because they are underperforming. Institutions may adopt new technologies because they have stronger resources.
These characteristics may influence outcomes independently of the intervention.
Statistical adjustment for measured covariates can sometimes reduce confounding, but it does not guarantee comparability on unmeasured factors.
Researchers should investigate the treatment assignment process rather than assuming that a large sample or sophisticated regression model resolves selection bias.
When a natural experiment may offer stronger identification
Some interventions arise from external events, administrative rules, or policy changes that generate plausibly exogenous variation in exposure.
For example, a policy may apply to institutions on different sides of an eligibility threshold or be introduced according to an externally determined schedule.
Such circumstances may support a natural-experimental argument if the assignment mechanism is sufficiently independent of potential outcomes.
However, an intervention is not automatically a natural experiment merely because researchers did not assign it.
The distinction between natural experiments and quasi-experiments depends partly on the terminology used and the nature of the identifying variation. The categories can overlap.
Why quasi-experimental research can support causal inference
Randomization is an important method for causal identification, but it is not the only possible source of credible causal evidence.
Regression discontinuity, difference-in-differences, instrumental-variable methods, and interrupted time-series designs can identify causal effects under their respective assumptions.
The strength of evidence depends on whether those assumptions are plausible, whether the intervention and outcome are measured appropriately, and whether alternative explanations have been addressed.
A well-designed quasi-experiment may provide more credible causal evidence than a poorly implemented randomized trial with serious allocation, attrition, or measurement problems.
Nevertheless, quasi-experimental estimates are not assumption-free. Their interpretation should reflect the specific population, treatment contrast, and identifying variation.
Why sample size does not compensate for weak identification
Large datasets can produce highly precise estimates of associations that are nevertheless causally biased.
Suppose administrative records contain outcomes for 100,000 students. If treatment assignment depends on unmeasured motivation and institutional resources, a narrow confidence interval does not establish that the estimated difference is causal.
Statistical precision concerns uncertainty around an estimator under its assumptions. Causal identification concerns whether that estimator represents the intended intervention effect.
Researchers should therefore prioritize the credibility of the comparison before emphasizing participant count or statistical significance.
Why the estimand must be specified
A quasi-experimental design should identify the causal effect it aims to estimate.
This may be an average treatment effect for treated institutions, a local effect near an eligibility threshold, or a change in outcome level following a policy intervention.
Different designs may identify different effects even when evaluating the same program.
For example, regression discontinuity may estimate the effect of tutoring eligibility for students near a diagnostic cutoff, while difference-in-differences may estimate an average effect among schools adopting the program.
Researchers should not assume that these estimates answer identical policy questions.
When a quasi-experimental design may be inappropriate
A quasi-experimental label is not justified as a strong causal strategy when no credible comparison or identifying feature exists.
For example, a single post-intervention survey asking participants whether a program helped them may provide useful perceptions but cannot independently establish its causal effect.
Similarly, comparing intervention participants with a convenient untreated group may be inadequate when selection mechanisms are poorly understood and important confounders cannot be addressed.
In such circumstances, descriptive, longitudinal observational, mixed-methods, or implementation research may better represent what the available evidence can support.
Choosing a more modest design label is preferable to overstating the strength of causal inference.
How to select the most defensible quasi-experimental strategy
Begin with the intervention assignment mechanism. Determine whether treatment was introduced through a cutoff, a policy change, a staggered rollout, or another identifiable process.
Next, examine the available data and the counterfactual assumptions each candidate design would require.
Then consider the primary estimand, outcome timing, participant composition, and plausible threats to validity.
The preferred design is the one whose assumptions can be defended most credibly using substantive knowledge and available evidence, not necessarily the one associated with the most advanced statistical technique.
The broader process of choosing among quasi-experimental designs should therefore begin with identification rather than software availability.
