How Do You Choose Among Quasi-Experimental Designs?

Which Quasi-Experimental Design Best Fits Your Research Question?

You want to evaluate whether an intervention caused a change in an outcome, but participants were not randomly assigned to receive it. Perhaps a university introduced an academic advising policy, a government allocated funding according to an eligibility threshold, or some schools adopted an educational technology program earlier than others.

Several quasi-experimental designs might appear suitable. Difference-in-differences compares changes across groups. Regression discontinuity exploits assignment thresholds. Interrupted time series examines changes in outcomes over time. Instrumental-variable approaches use a source of treatment variation that may be independent of the factors confounding treatment selection.

The difficulty is that these methods do not identify causal effects under the same conditions. Each depends on a particular source of information and a set of assumptions about what would have happened without the intervention.

Choosing among them therefore requires more than asking which method is familiar, statistically sophisticated, or commonly used in published research. The design must correspond to a credible causal identification opportunity.

Choose the Design That Best Exploits a Defensible Source of Causal Identification

What Should Determine Your Choice of Quasi-Experimental Design?

Start With the Causal Question, Not the Statistical Method

Before selecting a design, define the intervention or exposure, outcome, target population, and causal effect you want to estimate.

For example, asking whether students who use an artificial intelligence tutoring system perform better than nonusers is not equivalent to asking whether offering access to the system improves achievement. The first question concerns actual use; the second concerns the effect of access or assignment.

These effects may differ because some students offered access never use the system, while students who voluntarily use it may differ from nonusers in motivation, prior achievement, or available resources.

Researchers should therefore distinguish treatment assignment from treatment receipt and specify whether the target is an average treatment effect, an effect among treated participants, a local effect near an assignment threshold, or another estimand.

A design that identifies a narrow but credible effect may be preferable to one that appears to address the entire population but relies on implausible assumptions.

How Was Treatment Actually Assigned?

The treatment assignment mechanism is often the most informative starting point for design selection.

Researchers should investigate who determined treatment eligibility, whether an explicit rule was followed, whether adoption occurred at different times, and whether participation depended on individual or institutional choices.

Consider four situations:

• A scholarship is awarded when an eligibility score exceeds a predetermined threshold.
• A policy is introduced in one region while comparable regions remain unexposed.
• A university implements a new system on a particular date and has extensive historical outcome records.
• A randomized encouragement changes participation in an optional intervention without determining participation completely.

Although all four situations involve nonrandom treatment receipt, they offer different identification opportunities. The first may support regression discontinuity, the second difference-in-differences, the third interrupted time series, and the fourth instrumental-variable estimation.

These are possibilities rather than automatic classifications. Each requires additional assumptions and evidence.

What Is the Counterfactual, and Where Will It Come From?

A causal effect compares an observed outcome under one treatment condition with the outcome that would have occurred under another condition. For a particular unit, both potential outcomes generally cannot be observed simultaneously.

Quasi-experimental designs attempt to construct a credible counterfactual using features of treatment assignment, comparison populations, historical observations, or external variation.

The central question is not simply whether a comparison exists. It is whether the comparison provides defensible information about the outcome that would have occurred without treatment.

Different designs approach this problem differently.

These estimands are illustrative rather than exhaustive. Their precise interpretation depends on the assignment mechanism, treatment heterogeneity, model specification, and identifying assumptions.

When Should You Choose Difference-in-Differences?

Difference-in-differences (DiD) may be appropriate when treatment affects one group but not another, and outcomes are observed before and after intervention.

The approach compares how outcomes change in the treated group with how they change in the comparison group. Its causal interpretation commonly relies on a parallel-trends assumption: without treatment, the relevant average outcomes would have changed similarly across groups, potentially conditional on covariates.

Suppose one school district adopts a digital assessment platform while another does not. If both districts would plausibly have experienced similar achievement trends without adoption, their outcome changes may provide information about the platform's effect.

However, if the adopting district also introduces major curriculum reforms, or if the districts face different changes in student composition, the comparison may be confounded.

Researchers should investigate whether the parallel-trends assumption is plausible, rather than assuming that similar baseline scores establish it.

Staggered treatment adoption introduces additional complications. Conventional two-way fixed-effects estimators can produce difficult-to-interpret weighted comparisons when treatment effects vary across groups or over time. Appropriate modern DiD estimators should be considered for such settings.

When Should You Choose Regression Discontinuity?

Regression discontinuity (RD) may be appropriate when treatment assignment changes at a clearly defined threshold of a continuously measured or sufficiently granular running variable.

For example, students scoring below a specified diagnostic threshold may qualify for remedial instruction, while those immediately above it do not.

If relevant potential-outcome relationships are continuous at the cutoff and the assignment process is not compromised by precise manipulation or other discontinuities, students immediately around the threshold may provide a credible local comparison.

RD can be particularly persuasive when the assignment rule is documented and consistently implemented.

Its principal limitation is often external validity. The identified effect generally concerns individuals near the threshold, not necessarily all eligible students.

Before selecting RD, determine whether the assignment cutoff genuinely supports a discontinuity-based comparison. A threshold used merely for reporting or categorization may not provide the required identification opportunity.

When Should You Choose Interrupted Time Series?

Interrupted time series (ITS) may be suitable when an intervention begins at an identifiable time and repeated outcome observations are available before and after implementation.

It is particularly useful for institution-wide or population-wide interventions that cannot readily be assigned to untreated groups.

For example, a university introduces a mandatory academic advising policy and maintains monthly withdrawal records over several years. ITS can examine whether the withdrawal trajectory changes around implementation.

The design requires a defensible model of what would have happened without the intervention. Concurrent events, seasonal patterns, autocorrelation, and changes in outcome measurement may undermine the analysis.

The number of observations should be justified according to the expected effect and temporal structure, rather than a universal minimum. Researchers should assess whether the available series is sufficiently informative for the planned analysis.

When a credible comparison series exists, controlled ITS may help account for broader temporal influences.

When Should You Choose an Instrumental-Variable Design?

An instrumental-variable (IV) strategy may be useful when treatment selection is endogenous but a credible source of variation affects treatment without independently influencing the outcome through prohibited pathways.

For example, randomized encouragement to participate in optional tutoring might serve as an instrument for actual participation if it changes uptake and satisfies the relevant independence and exclusion assumptions.

IV estimation can address certain unmeasured-confounding problems, but instrument validity is demanding. A variable that strongly predicts treatment is not necessarily a valid instrument.

The resulting effect may also apply to a particular subgroup, such as participants whose treatment status changes because of the instrument.

The decision to use IV should therefore depend on whether the proposed instrument can support the intended causal inference, not simply on the presence of endogeneity.

When Is a Controlled Interrupted Time Series Preferable to Difference-in-Differences?

Both approaches can use treated and untreated populations observed over time, but their analytical emphasis and assumptions may differ.

A conventional two-period DiD analysis compares changes between two periods. Controlled ITS generally uses a longer series to model intervention-associated changes in level, slope, or another temporal response.

Controlled ITS may be particularly useful when the intervention is expected to change the outcome trajectory and enough repeated observations exist to characterize that change.

DiD may be more appropriate when the principal estimand is an average post-treatment effect and the untreated comparison group's outcome changes provide a credible counterfactual under the specified design.

Neither method is categorically superior. Extended DiD and comparative time-series approaches may overlap substantially, and both require careful attention to concurrent events and treatment-effect heterogeneity.

Can a Naturally Occurring Policy Change Determine the Design?

A naturally occurring policy change can create a useful identification opportunity, but calling it a natural experiment does not specify how the causal effect will be identified.

A policy introduced at an eligibility cutoff may support RD. A policy affecting one jurisdiction but not another may support DiD. A nationwide policy implemented on a particular date may support ITS.

Whether these opportunities are credible depends on the assignment mechanism and counterfactual assumptions.

Researchers should not confuse the source of variation with the analytical design. A natural experiment describes features of how exposure arose, whereas RD, DiD, ITS, and IV describe particular identification or estimation strategies.

What If More Than One Design Appears Suitable?

Some research settings support multiple plausible strategies. A policy may have an eligibility threshold, be introduced at different times across regions, and generate repeated outcome measurements.

Researchers should first determine which strategy most directly exploits the strongest identification feature.

For example, a well-enforced assignment threshold may provide a persuasive local comparison even if the data also support a broader before-and-after analysis.

Additional methods can sometimes serve as robustness checks or identify complementary estimands. However, agreement between methods does not automatically prove causality, particularly when they share important sources of bias.

Researchers should specify why each method is informative, what effect it estimates, and which assumptions differ.

What Data Requirements Should Influence Design Selection?

Each design requires data aligned with its identification strategy.

Data availability matters, but it should not reverse the logic of design selection. A dataset may contain the variables needed to calculate an estimator without containing the information needed to justify its causal interpretation.

How Should Statistical Power and Precision Affect the Decision?

A credible identification strategy must also be statistically informative.

RD estimates may be imprecise when relatively few observations lie near the cutoff. IV estimates may be unstable when instruments are weak. ITS estimates may be poorly determined when the series is short or strongly autocorrelated.

DiD estimates may be unreliable when few independent treatment-assignment units exist or when the dependence structure is ignored.

Researchers should therefore evaluate statistical power, expected confidence-interval width, clustering, and model-specific uncertainty before committing to a design.

Nevertheless, greater precision does not compensate for invalid identification assumptions. A precisely estimated biased effect remains biased.

Can a More Sophisticated Design Compensate for Poor Implementation?

No. The credibility of quasi-experimental research depends on how the design operates in the actual study.

A regression discontinuity design is weakened if participants manipulate their assignment scores. DiD is threatened if comparison groups experience different untreated trends. ITS may be confounded by another policy introduced at the same time. IV fails to identify the intended effect if the proposed instrument influences the outcome through an alternative pathway.

These problems cannot be resolved merely by using more elaborate regression models.

The relevant standard is whether the research design provides a credible explanation for why the observed comparison approximates the missing counterfactual.

Choosing a Design to Evaluate an AI Tutoring Program

Suppose the university uses a strictly enforced diagnostic cutoff and has extensive outcome data immediately around that threshold. RD may provide a particularly credible estimate for students near the cutoff.

However, if policymakers want to know the effect of offering tutoring to all students, that local estimate may not answer the entire policy question. Additional evidence would be needed to assess generalizability.

The example demonstrates why design selection involves both identification credibility and alignment between the estimand and the decision the research is intended to inform.

Common Mistakes When Selecting a Quasi-Experimental Design

A Practical Decision Framework for Choosing Your Design

The most useful first step is to reconstruct the intervention assignment process. This may require examining policy documents, implementation records, institutional decisions, or administrative procedures before selecting an estimator.

Once the assignment mechanism is understood, identify the counterfactual comparison it makes possible and evaluate the assumptions that comparison requires.

Document the reasoning before conducting the primary analysis. A defensible research protocol should specify the treatment mechanism, target estimand, identification assumptions, anticipated threats, and planned sensitivity analyses.

If several designs remain plausible, consider whether they identify complementary effects or provide meaningful sensitivity checks. Do not combine methods simply to make the analysis appear more sophisticated.

The most appropriate design is the one whose causal argument can survive careful examination, including the possibility that the evidence will not support the desired conclusion.

Before Selecting a Quasi-Experimental Design, Verify These Conditions

Frequently Asked Questions About Choosing Quasi-Experimental Designs

The Best Quasi-Experimental Design Is the One With the Most Credible Counterfactual

How to Cite This Guide

This guide is intended to be read, shared, and used in research, teaching, and academic work. If you draw on its ideas, explanations, or other content, please acknowledge the source by citing the guide. Doing so gives appropriate credit and helps your readers locate the original resource.