Can a Quasi-Experiment Support Causal Inference?

Can You Claim That an Intervention Caused an Outcome When Participants Were Not Randomly Assigned?

Suppose a university introduces an AI-supported academic advising system to several campuses. Student retention improves after implementation, and campuses using the system outperform campuses that have not adopted it.

You want to conclude that the advising system improved retention. However, administrators selected the intervention campuses according to their technical readiness, and the campuses may differ in staffing, student characteristics, and institutional resources.

Does the absence of randomization mean that causal inference is impossible? Or can a quasi-experimental design provide sufficiently credible evidence that the intervention produced the observed improvement?

The answer depends on whether the study can distinguish the intervention's effect from alternative explanations. Randomization is one way to establish a credible comparison, but it is not the only possible basis for causal identification.

Quasi-Experiments Can Support Causal Inference Under Defensible Assumptions

What Makes Causal Inference Possible Without Random Assignment?

Causal inference requires a counterfactual comparison

A causal effect concerns what changes because an intervention occurs rather than what merely happens afterward.

Under the potential outcomes framework, each unit has outcomes associated with different treatment conditions. For a binary intervention, these are commonly represented as the outcome under treatment and the outcome without treatment.

For the same unit at the same relevant time, researchers ordinarily observe only the outcome corresponding to the treatment actually received. The alternative outcome is counterfactual.

For example, a university implementing an AI advising system has an observed retention rate under implementation. Researchers cannot simultaneously observe the retention rate that the same university would have experienced without the system.

The causal problem is to estimate that missing outcome using other observations and defensible assumptions.

Quasi-experimental methods differ in how they construct or identify the counterfactual. The fundamental causal problem remains the same.

Randomization is valuable, but it is not the definition of causality

Random assignment can make treatment independent of potential outcomes in expectation, providing a strong basis for estimating causal effects.

However, randomization is a design mechanism rather than a universal prerequisite for causal knowledge.

Policies, eligibility rules, administrative thresholds, and intervention timing can sometimes create comparisons that support causal identification without researcher-controlled random assignment.

For example, an educational support program assigned using a strict diagnostic-score cutoff may permit a credible local comparison between students immediately above and below that cutoff.

Similarly, a policy introduced at a known time may be evaluated using repeated outcome measurements and an appropriate comparison series.

These designs can support causal inference, but their validity depends on assumptions that differ from those of a randomized trial.

Identification and estimation are not the same

Causal identification concerns whether the target causal effect can be expressed in terms of observable data under specified assumptions.

Estimation concerns how the identified quantity is calculated from a finite dataset.

A researcher may estimate a regression coefficient with great numerical precision even when the coefficient does not identify the intended causal effect.

For example, a comparison between voluntary users and nonusers of an AI tutoring platform may produce a statistically significant achievement difference. If motivated students are more likely to use the platform, the coefficient may partly reflect motivation rather than the intervention.

Increasing the sample size can reduce sampling uncertainty without eliminating that systematic problem.

Researchers should therefore establish an identification argument before choosing an estimator or emphasizing statistical significance.

Different quasi-experimental designs rely on different assumptions

There is no single set of assumptions that makes every quasi-experiment causal.

Each design exploits a particular source of treatment variation and requires conditions under which that variation supports a valid comparison.

These assumptions are not interchangeable. Evidence supporting parallel trends does not establish the validity of an instrumental variable, and a credible regression discontinuity cutoff does not justify extrapolation to every member of the population.

How difference-in-differences can identify an intervention effect

Difference-in-differences compares changes in outcomes between treated and comparison groups.

Suppose retention increases by eight percentage points at universities introducing an AI advising program and by three percentage points at comparison universities.

The difference in changes is five percentage points.

That contrast can identify an average treatment effect for an appropriately defined treated population if the comparison group's outcome change represents the counterfactual change the treated group would have experienced without intervention, together with the other required assumptions.

However, if treated universities also introduced a financial assistance program, the five-point contrast may reflect both interventions.

Repeated pre-intervention observations can help assess the plausibility of the identifying assumptions, but observed pre-trends cannot prove what would have happened after treatment.

The parallel trends assumption is therefore a substantive causal requirement rather than a routine statistical test.

How regression discontinuity can support local causal inference

Regression discontinuity uses an assignment rule based on a continuous or ordered running variable and a specified cutoff.

For example, students scoring below 60 on a diagnostic assessment may become eligible for AI-supported tutoring, while students scoring at or above 60 do not.

Under suitable assumptions, students immediately around the threshold can provide a credible local comparison.

In a sharp regression discontinuity design, treatment assignment changes deterministically at the cutoff. In a fuzzy design, the probability of treatment changes discontinuously rather than moving from zero to one.

The causal estimand is generally local to the threshold. For fuzzy designs, additional instrumental-variable-type assumptions are needed for the usual local treatment-effect interpretation.

The design's credibility depends on the assignment mechanism, continuity assumptions, and whether the running variable can be precisely manipulated.

A credible cutoff can be more important than a large sample containing students far from the threshold.

How interrupted time series can support causal inference

Interrupted time-series analysis uses repeated observations to estimate whether an outcome changes following an intervention introduced at an identifiable time.

For example, a university may have monthly advising completion data for several years before and after adopting a new platform.

Researchers can model the pre-intervention outcome trajectory and examine whether the post-intervention series departs from the expected counterfactual.

However, the design is vulnerable when another event occurs near the intervention date or when the assumed pre-intervention trend would not have continued.

Seasonality, autocorrelation, measurement changes, and delayed intervention effects also require attention.

A credible comparison series can strengthen the design by providing information about background changes, although the comparison must itself satisfy appropriate assumptions.

Why selection bias threatens nonrandomized causal comparisons

In many quasi-experiments, intervention recipients differ systematically from nonrecipients.

Schools may be selected because of poor performance, strong infrastructure, or administrative readiness. Students may enter support programs because of prior achievement or perceived need.

These characteristics can influence outcomes independently of the intervention.

For example, universities with strong digital infrastructure may adopt an AI advising system earlier and may also have better retention because of existing institutional support.

Comparing their retention with that of later adopters may confound intervention effects with pre-existing institutional advantages.

The relevant question is whether the identification strategy addresses the particular selection mechanism, not whether the analysis includes many control variables.

Statistical adjustment cannot automatically remove unmeasured confounding

Regression adjustment, matching, and propensity-score methods may improve comparability on observed characteristics.

However, causal identification through these methods typically requires assumptions about the absence of relevant unmeasured confounding, together with appropriate overlap, measurement, and model conditions.

If motivation influences both voluntary AI tutoring use and achievement but is not adequately measured, adjustment for age, sex, and prior grades may not eliminate confounding.

Similarly, matching institutions on enrollment and geographic region does not guarantee comparability in leadership, institutional culture, or unmeasured policy changes.

Researchers should avoid treating covariate adjustment as a substitute for a credible design.

Why time-related confounding matters

Quasi-experimental evaluations frequently examine interventions introduced at particular times.

Outcomes may change independently because of maturation, economic conditions, institutional reforms, seasonal patterns, or other events.

For example, a digital learning platform introduced before final examinations may coincide with improvements arising from ordinary examination preparation.

In staggered implementation, early and late adopters may also follow different outcome trajectories because of the factors determining adoption timing.

Researchers should distinguish intervention effects from these background processes through appropriate design and analysis.

The effects of time-related confounding in rollout evaluations are particularly important when adoption timing is administratively determined.

Why the causal estimand must be explicit

Different quasi-experimental designs may identify different causal quantities.

A regression discontinuity design may identify a local effect for students near an eligibility cutoff. Difference-in-differences may identify an average effect among treated institutions under its assumptions.

An interrupted time-series analysis may estimate a change in outcome level or slope relative to a modeled counterfactual trajectory.

These effects are not necessarily interchangeable.

Researchers should specify the population, intervention, comparator, outcome, and effect measure before selecting an analytical strategy.

They should also distinguish effects of intervention assignment, eligibility, actual receipt, and sustained exposure where these differ.

Internal validity and generalizability are separate questions

A quasi-experiment may provide credible causal evidence for a narrowly defined population without establishing that the effect applies elsewhere.

For example, a regression discontinuity study may identify the effect of tutoring eligibility among students near a diagnostic threshold.

That estimate does not automatically establish the effect among students with substantially higher or lower scores.

Similarly, an intervention evaluated in technically prepared universities may produce different effects in institutions with limited infrastructure.

Researchers should distinguish whether the estimated effect is internally credible from whether it can be transported or generalized to other populations and settings.

Why statistical significance is not evidence of causal identification

A statistically significant association can arise from confounding, selection, or measurement bias.

A very large sample may produce a narrow confidence interval around an estimate that is systematically biased.

Conversely, a credible quasi-experiment may produce an imprecise estimate because the available sample or identifying variation is limited.

Statistical significance addresses uncertainty under an estimator and its assumptions. It does not independently establish that the estimator identifies a causal effect.

Researchers should evaluate effect magnitude, uncertainty, identifying assumptions, and alternative explanations together.

What role do robustness and falsification analyses play?

Robustness analyses examine whether conclusions change under plausible alternative specifications or assumptions.

For example, a regression discontinuity analysis may examine different bandwidths and functional forms. An interrupted time-series study may assess alternative trend specifications or intervention dates.

Falsification analyses may examine outcomes or periods where the intervention should not plausibly produce an effect.

Negative control outcomes or exposures can sometimes help detect particular forms of confounding.

However, passing robustness checks does not prove that all assumptions hold. Different models may share the same underlying bias.

Such analyses should be selected according to the design's plausible threats and, where possible, prespecified before outcome inspection.

When causal language is justified

Causal language can be appropriate when a quasi-experimental design has a clear estimand, credible identification strategy, suitable data, and a transparent assessment of assumptions.

For example, researchers may state that a policy increased an outcome under the identifying assumptions of a regression discontinuity or difference-in-differences design.

However, the language should reflect the scope and uncertainty of the evidence.

When major assumptions are weak or unverifiable with the available information, more qualified statements may be appropriate.

The objective is neither to prohibit causal language in nonrandomized research nor to use it automatically whenever an estimator is described as quasi-experimental.

When a quasi-experiment cannot credibly support causality

A study may lack sufficient evidence for causal inference when treatment assignment is strongly related to unmeasured outcome determinants, no credible comparison exists, or major concurrent changes cannot be separated from the intervention.

For example, a university introducing an AI advising system and a new financial assistance program on the same date may be unable to identify the separate effect of advising without additional variation or assumptions.

Similarly, a one-group pretest-posttest study may not distinguish intervention effects from ordinary outcome changes.

In these circumstances, descriptive or associational conclusions may be more defensible.

The distinction between a weak quasi-experiment and a strong observational study should be based on identification quality rather than methodological labels.

Can a Nonrandomized AI Advising Evaluation Support a Causal Conclusion?

Interpretation: The observed difference-in-differences is five percentage points, but the arithmetic alone does not establish causality.

If technical readiness is associated with different counterfactual retention trends, the estimate may be biased. If the campuses have credible comparable trends and no important differential concurrent changes, a causal interpretation becomes more defensible.

The central question is whether the comparison campuses approximate what would have happened at intervention campuses without the advising system.

Common Misconceptions About Causal Inference in Quasi-Experiments

Build the Causal Argument Before Estimating the Effect

Begin by defining the intervention effect you want to estimate. Then investigate how treatment was assigned and identify the design feature that could support a credible counterfactual.

Specify the assumptions required for identification and determine what evidence can be used to evaluate their plausibility.

Only then should you select the statistical estimator and plan robustness analyses.

A credible quasi-experiment explains why the estimated contrast represents an intervention effect. The statistical model operationalizes that argument; it does not create the argument by itself.

Before Making a Causal Claim From a Quasi-Experiment

Frequently Asked Questions About Quasi-Experimental Causality

Quasi-Experiments Can Identify Causal Effects, but Their Assumptions Carry the Argument

How to Cite This Guide

This guide is intended to be read, shared, and used in research, teaching, and academic work. If you draw on its ideas, explanations, or other content, please acknowledge the source by citing the guide. Doing so gives appropriate credit and helps your readers locate the original resource.