03 · What You Need to Know
How to Evaluate the Credibility of a Quasi-Experimental Comparison Group
The comparison group represents an unobserved alternative
In causal research, the outcome that a treated individual or institution would have experienced without treatment is counterfactual.
Researchers cannot ordinarily observe both treatment and non-treatment outcomes for the same unit at the same relevant time.
A comparison group provides information that may help estimate the missing outcome.
For example, universities that did not adopt an AI advising system may provide evidence about retention changes that could have occurred without the intervention.
However, the untreated universities are not automatically equivalent to the treated universities.
The credibility of the comparison depends on whether the design's assumptions justify using their outcomes to approximate the treated group's counterfactual.
Why baseline similarity is not enough
Researchers often select comparison institutions based on observable characteristics such as enrollment, geographic region, institutional type, or baseline achievement.
These characteristics may be useful because they are associated with outcomes or intervention adoption.
However, similarity on a limited set of observed variables does not guarantee similarity in unobserved determinants of outcomes.
Two schools may have nearly identical baseline achievement but differ in teacher turnover, instructional leadership, parental involvement, or planned curriculum reforms.
If those differences influence subsequent achievement, the schools may follow different trajectories without the intervention.
Consequently, a comparison group should be assessed according to the identification assumptions of the design rather than a generic requirement that baseline means be similar.
What makes a comparison credible depends on the design
Different quasi-experimental designs require different forms of comparability.
Difference-in-differences relies on assumptions about counterfactual outcome changes. Regression discontinuity relies on local continuity around an assignment cutoff. Interrupted time-series designs rely on assumptions about how outcomes would have evolved without intervention.
| Design |
Relevant comparison |
Key credibility question |
| Difference-in-differences |
Treated and untreated or not-yet-treated groups observed over time |
Would their outcomes have changed comparably without treatment, under the specified assumptions? |
| Regression discontinuity |
Units immediately around an assignment cutoff |
Are relevant potential outcomes continuous at the threshold apart from the treatment discontinuity? |
| Controlled interrupted time series |
Repeated treated and comparison outcome series |
Does the comparison series represent relevant background changes without being affected by treatment? |
| Matching-based observational causal analysis |
Units with similar measured pretreatment covariates |
Are relevant confounders adequately measured, and is there sufficient overlap? |
| Staggered rollout evaluation |
Appropriate not-yet-treated or never-treated units |
Is adoption timing compatible with the required counterfactual assumptions? |
A comparison suitable for one design may be inappropriate for another. Researchers should therefore identify the causal strategy before declaring that the comparison group is credible.
Why the treatment assignment mechanism matters
The first question should be why some units received the intervention and others did not.
Was treatment assigned according to eligibility, administrative readiness, voluntary participation, geographic location, or policy timing?
These mechanisms can create systematic differences between treated and comparison units.
For example, universities adopting an AI advising system may have stronger digital infrastructure and more experienced administrative personnel.
Those characteristics may independently influence retention.
Researchers should investigate whether the comparison group differs on factors that determine both treatment receipt and potential outcomes.
The allocation mechanism often provides more information about causal credibility than a table of demographic similarities alone.
Why comparable outcome trajectories may matter more than identical baseline levels
In difference-in-differences research, the central identifying assumption concerns counterfactual outcome trends rather than equal outcome levels.
Two universities may have different baseline retention rates but similar changes over time in the absence of treatment.
Conversely, two universities may have identical baseline retention rates while following different underlying trajectories.
The latter situation can be particularly problematic for a difference-in-differences analysis.
Consider two hypothetical universities with the following pre-intervention retention rates:
| University |
Year 1 |
Year 2 |
Year 3 |
| University A |
70% |
75% |
80% |
| University B |
78% |
79% |
80% |
Both universities have 80% retention in Year 3, but University A improved by five percentage points annually while University B improved by one.
If University A adopts an intervention after Year 3, University B's future outcome trajectory may not provide a credible counterfactual without additional justification.
This example demonstrates why matching on the latest baseline outcome alone can be misleading.
Nevertheless, observed pre-intervention trends cannot prove the parallel trends assumption for the unobserved post-intervention period.
How eligibility criteria affect comparison credibility
Treated and comparison populations should be defined using eligibility criteria appropriate to the intended estimand.
Suppose an AI tutoring intervention is offered only to first-year students who fail a diagnostic assessment.
Comparing these students with all students who did not receive tutoring would combine differences in treatment exposure with differences in academic need and eligibility.
A more defensible comparison might involve students subject to the same eligibility rules but not receiving treatment for reasons compatible with a particular identification strategy.
In regression discontinuity, for example, students immediately around an eligibility threshold may provide a local comparison under continuity assumptions.
Researchers should define who could have received the intervention and why some eligible units remained untreated.
Why common measurement procedures matter
A comparison group may be similar in relevant characteristics but still be unsuitable if outcomes are measured differently.
For example, intervention schools may use a new digital assessment while comparison schools use a conventional examination.
Differences in observed scores may then reflect assessment format or measurement properties rather than the intervention.
Researchers should examine whether outcome definitions, measurement instruments, assessment schedules, and data collection procedures are sufficiently comparable.
Changes in administrative reporting or participant eligibility can also create artificial differences over time.
Measurement comparability is therefore part of counterfactual credibility, not merely a data management concern.
Why contamination and spillover can undermine comparison groups
Comparison units should not be assumed untreated simply because they were not formally assigned the intervention.
Students may share digital learning materials across classes. Teachers may adopt instructional practices learned from colleagues in intervention schools. Institutional policies may influence nearby campuses.
Such spillover can change comparison outcomes and complicate interpretation.
For example, if teachers in comparison schools receive informal training from intervention teachers, their students may benefit from elements of the intervention.
The resulting estimate may represent the effect of formal program assignment under spillover rather than the effect of exposure in a setting without interference.
Researchers should identify plausible contamination pathways and determine whether the intended estimand remains meaningful.
Why concurrent interventions are particularly problematic
Suppose intervention universities introduce an AI advising system while simultaneously expanding financial assistance.
If comparison universities do not receive the financial assistance, differences in retention may reflect both changes.
Researchers should distinguish the intervention being evaluated from other policies introduced at the same time.
If the policies always occur together, the data may support estimation of a combined package effect rather than the separate effect of the advising system.
Calendar-time adjustment alone cannot necessarily separate interventions that are perfectly aligned across institutions and periods.
Credible comparison selection therefore requires understanding institutional policy histories, not simply matching baseline characteristics.
How matching can improve comparison quality
Matching attempts to identify treated and untreated units with similar measured pretreatment characteristics.
Researchers may match schools on baseline achievement, enrollment, geographic region, or other prognostic factors.
Propensity-score methods summarize the estimated probability of treatment conditional on measured covariates and can support particular matching or weighting strategies.
However, matching does not automatically create randomization.
Its causal interpretation depends on assumptions about measured confounding, overlap, treatment definition, and the analysis.
Researchers should not select comparison units solely because a matching algorithm produces small numerical differences on observed variables.
Why overlap matters
Overlap, sometimes called positivity, concerns whether units with relevant covariate profiles have a meaningful possibility of receiving either treatment condition.
Suppose every large university adopts an AI advising system and every small university remains untreated.
The dataset may contain no untreated large universities from which to estimate the counterfactual outcome for large treated universities without substantial extrapolation.
Statistical models can produce predictions in such settings, but those predictions may depend heavily on assumptions unsupported by comparable observations.
Researchers should examine the distribution of relevant characteristics and identify regions where treated and comparison units overlap.
Restricting analysis to an overlap population may improve credibility but can change the target estimand and limit generalizability.
Why comparison groups should be selected before examining treatment effects
Researchers may be tempted to compare several possible untreated groups and retain the one producing the strongest or most statistically significant intervention effect.
This introduces analytical flexibility and can undermine the credibility of the findings.
Comparison selection should be guided by the assignment mechanism, substantive knowledge, pretreatment characteristics, and design assumptions.
Where feasible, researchers should prespecify the primary comparison and criteria for including or excluding units.
Alternative comparison groups may be used in sensitivity analyses, but their selection and interpretation should be transparent.
A credible comparison should be chosen because it supports the counterfactual, not because it produces a desirable result.
Why post-treatment variables should not automatically be used for matching
Matching variables should generally be measured before intervention exposure when they are intended to address baseline confounding.
Variables measured after treatment may themselves be affected by the intervention.
For example, student engagement after implementation may partly reflect the effect of an AI tutoring platform.
Matching intervention and comparison students on post-treatment engagement could remove part of the effect researchers seek to estimate or introduce bias through conditioning on a consequence of treatment.
Researchers should distinguish pretreatment confounders from mediators, colliders, and other post-treatment variables using substantive causal reasoning.
When not-yet-treated units can serve as comparisons
In staggered intervention rollouts, institutions scheduled to adopt the program later may serve as comparisons for institutions that have already adopted it.
However, not-yet-treated status alone does not establish comparability.
Early and late adopters may differ in readiness, resources, anticipated outcomes, or institutional priorities.
Researchers should examine whether the required counterfactual trends are plausible and whether anticipation or preparatory activities affect outcomes before formal implementation.
Once later adopters receive the intervention, they no longer provide untreated concurrent observations.
Modern staggered difference-in-differences approaches can define appropriate comparison sets and treatment-effect aggregation strategies, but their causal validity remains conditional on identification assumptions.
When a comparison series is useful in interrupted time-series research
Controlled interrupted time-series designs use repeated outcomes from a comparison population or outcome series alongside the treated series.
A useful comparison series should capture relevant background changes without being affected by the intervention.
For example, a comparable university that did not introduce an advising policy may help identify broader changes in retention occurring during the same period.
However, the comparison university may experience different policy changes or student population shifts.
Researchers should examine the similarity of prior trends, measurement consistency, exposure status, and concurrent events.
The advantages of controlled interrupted time-series designs depend on the credibility of the control, not simply its presence.
How negative controls can help assess comparison credibility
Negative controls are outcomes or exposures that, under a specified causal argument, should not be affected by the intervention but may reveal particular sources of bias.
For example, researchers might examine an outcome that should not plausibly respond to the educational intervention but is subject to similar institutional confounding.
An unexpected association could suggest that the comparison remains affected by factors other than treatment.
However, choosing a valid negative control requires substantive justification. An outcome assumed unaffected may actually respond indirectly to the intervention.
Negative control findings can strengthen or challenge an identification argument, but they do not automatically establish that all confounding has been removed.
What if no credible comparison group exists?
Some interventions are introduced in ways that make a credible untreated comparison difficult to construct.
For example, every university in a system may adopt a new policy simultaneously, while other institutions differ substantially in student populations and reporting procedures.
Researchers may consider alternative identification strategies, including interrupted time series, regression discontinuity, or other approaches suited to the assignment mechanism.
However, an alternative design is useful only when its assumptions can be defended.
If no credible counterfactual can be established, the study may still provide descriptive, implementation, or associational evidence.
Researchers should avoid presenting an unsuitable comparison group as valid merely because the research question requires one.
Watch Out
A comparison group can be similar to the intervention group on every variable in a baseline table and still provide a poor counterfactual. Comparability must be evaluated in relation to the assignment mechanism, outcome trajectories, unmeasured factors, and assumptions of the chosen design.
