03 · What You Need to Know
What Makes a Natural-Experiment Comparison Group Credible?
What Is the Comparison Group Supposed to Accomplish?
The purpose of a comparison group is to provide information about the outcome that exposed units would have experienced under an alternative exposure condition.
This missing outcome is the counterfactual. Because researchers cannot ordinarily observe the same individual or institution simultaneously under both exposure conditions, they must rely on an identification strategy that connects observed comparison outcomes to the missing counterfactual.
For example, suppose schools receiving a connectivity subsidy increase their online learning participation by 15 percentage points. That improvement alone does not establish the subsidy effect. Participation might also have increased because of wider internet availability, changes in school requirements, or economic conditions.
A suitable comparison group helps determine how participation might have changed without the subsidy.
However, different designs require different kinds of comparability. A difference-in-differences analysis may rely on assumptions about untreated trends, while regression discontinuity relies on continuity near an assignment threshold. Researchers should therefore select comparisons according to the identification strategy rather than applying one universal matching rule.
Why Must You Understand How Exposure Was Assigned?
Comparison selection should begin with the process that determined which units received the intervention.
Suppose a government allocated digital devices to schools with the lowest achievement scores. Recipient schools may differ systematically from nonrecipients in resources, prior performance, or student characteristics.
Choosing higher-performing schools as controls simply because they did not receive devices could produce a misleading comparison.
Alternatively, suppose allocation depended on a predetermined eligibility threshold. Schools immediately around the cutoff may offer a more credible local comparison than schools selected because they resemble recipients on a broad collection of characteristics.
The researcher should document eligibility criteria, administrative discretion, implementation timing, and any opportunities for institutions to influence their exposure status.
Understanding how naturally occurring exposure variation was generated helps identify which comparison units are relevant and which differences could threaten causal inference.
Does the Comparison Group Need to Be Identical to the Exposed Group?
No. Exact similarity across every characteristic is neither feasible nor required by all causal designs.
What matters is whether differences between groups undermine the assumptions needed for the intended counterfactual comparison.
For example, two universities may have different baseline withdrawal rates but similar untreated changes over time. Under appropriate assumptions, they may still support a difference-in-differences analysis.
Conversely, two universities may have nearly identical baseline withdrawal rates but experience different trends because of student composition, institutional funding, or regional economic conditions.
The second pair may be less useful despite its apparent similarity at a single time point.
Researchers should distinguish baseline-level comparability from comparability in the potential outcomes or changes required by the chosen design.
Which Characteristics Should Be Considered When Selecting Controls?
Relevant characteristics are those that help explain exposure assignment, outcome variation, or the plausibility of the counterfactual.
In educational research, these may include prior achievement, enrollment patterns, socioeconomic composition, institutional resources, school size, location, and historical exposure to related interventions.
However, the appropriate variables depend on the causal question. A variable that is useful for predicting outcomes may not always be appropriate for matching or adjustment.
Researchers should generally prioritize characteristics measured before exposure. Matching on variables affected by the intervention can introduce bias or remove part of the effect being estimated.
A causal diagram or explicit substantive model can help identify which variables are relevant confounders, which are mediators, and which might create collider bias if conditioned upon.
The objective is not to maximize the number of matching variables. It is to improve the credibility of the intended comparison.
Why Are Pre-Intervention Outcome Trends Often Important?
Historical outcome patterns can reveal differences that a single baseline measurement conceals.
Suppose two school districts have the same attendance rate immediately before a policy begins. In one district, attendance has been increasing for several years. In the other, it has been declining.
Comparing their subsequent changes without accounting for these trajectories may produce a misleading estimate.
For difference-in-differences, the key concern is whether untreated outcome trends would have been comparable under the specified assumptions.
Multiple pre-intervention observations can help evaluate the plausibility of that assumption, although similar observed trends do not prove that the unobserved post-intervention trends would have remained parallel.
Researchers should also consider whether the pre-intervention period contains anticipation effects, unusual shocks, or changes in measurement.
Can Geographically Nearby Institutions Serve as Controls?
Geographic proximity may improve comparability in some settings because nearby institutions experience similar economic conditions, weather, administrative environments, or policy influences.
However, proximity does not establish counterfactual validity.
Neighboring schools may differ in admissions policies, socioeconomic composition, instructional resources, or technology readiness. They may also be affected by spillovers from the intervention.
For example, teachers in a subsidized district may share digital materials with teachers in adjacent districts. The neighboring district would then be partially exposed despite being classified as untreated.
Sometimes a more distant institution with comparable historical outcome dynamics and limited exposure to spillovers may provide a better comparison.
Geographic distance should therefore be treated as a contextual consideration rather than a sufficient selection criterion.
Can Not-Yet-Treated Units Be Used as Comparison Groups?
Yes, under appropriate conditions. When a policy is introduced at different times, units that have not yet received the intervention may provide comparison information for units already treated.
For example, a government may introduce a learning management system in several school districts during the first year and in others during the following year.
Later-adopting districts could potentially serve as comparisons before their own implementation.
However, the researcher must examine why some districts received the intervention earlier. Implementation timing may depend on readiness, resources, anticipated outcomes, or administrative priorities.
Anticipation also matters. Districts expecting the intervention may begin training or change procedures before formal implementation.
With staggered treatment adoption, researchers should use estimators appropriate to the timing structure and potential treatment-effect heterogeneity. Conventional two-way fixed-effects comparisons can be misleading when already-treated units contribute inappropriate counterfactual information.
Can Regression Discontinuity Provide a Comparison Without Conventional Matching?
Yes. In regression discontinuity, the comparison is constructed around a treatment-assignment threshold.
Suppose students with diagnostic scores below 50 qualify for additional academic support. Students scoring just below and just above the threshold may be comparable in relevant potential outcomes, even if students far from the threshold differ substantially.
The design relies on continuity assumptions and the integrity of the assignment process, rather than requiring broad similarity between all treated and untreated students.
Researchers should examine whether the threshold actually changes treatment assignment, whether individuals can manipulate the running variable, and whether other consequential policies change at the same cutoff.
The resulting estimate is generally local to the threshold.
These requirements are central to determining whether an assignment cutoff supports a credible regression discontinuity design.
Can Matching Improve Comparison-Group Selection?
Matching can help construct a comparison group with similar measured pre-exposure characteristics.
Researchers may use exact matching, propensity-score matching, or other balancing procedures, depending on the design and available data.
For example, schools receiving a digital infrastructure subsidy might be matched with nonrecipient schools based on pre-policy achievement, enrollment, resources, and geographic characteristics.
However, matching does not automatically remove unmeasured confounding. It also does not establish parallel trends, instrument validity, or the continuity assumptions required by other designs.
Researchers should evaluate balance after matching and determine whether sufficient overlap exists between treated and comparison units.
If treated schools have characteristics absent among untreated schools, the available data may not support reliable comparisons for the entire treated population.
When Might Synthetic Control Methods Be Useful?
Synthetic control methods may be useful when a policy affects one or a small number of aggregate units and several untreated units provide historical outcome information.
Rather than selecting one comparison institution, the method constructs a weighted combination of potential controls to approximate the treated unit's pre-intervention outcome trajectory and, where appropriate, other predictors.
For example, a university introducing a distinctive retention policy might be compared with a weighted combination of universities that did not adopt the policy.
The approach requires careful assessment of the donor pool, pre-intervention fit, possible spillovers, and assumptions supporting the post-intervention counterfactual.
A good historical fit is informative but does not guarantee that the synthetic comparison will remain valid after intervention.
Synthetic control is therefore an identification strategy requiring substantive justification, not simply an automated procedure for finding similar units.
How Do Spillovers Affect Comparison-Group Credibility?
A comparison group should represent the intended alternative exposure condition. Spillovers threaten this requirement when the intervention influences units classified as untreated.
In educational technology research, spillovers may occur through shared software accounts, instructional materials, professional development, or informal collaboration among teachers.
For example, a district receiving AI-assisted teaching resources may share lesson plans with neighboring districts. Their outcomes could improve partly because of the intervention, reducing the apparent difference between treated and untreated groups.
The direction of bias is not always predictable. Spillovers can alter behavior, resource allocation, or competition in ways that produce more complicated effects.
Researchers should investigate likely channels of interference and consider whether alternative comparison units or explicit spillover estimands are needed.
What If the Comparison Group Experiences a Different Concurrent Event?
A comparison population may experience changes that do not affect the exposed population.
Suppose the treated university introduces a retention policy while the comparison university simultaneously expands scholarships. Both interventions may influence withdrawal rates.
Subtracting the comparison university's change could then distort the estimated retention-policy effect.
Researchers should document concurrent events affecting both populations and evaluate whether those events are shared, differential, or potentially related to treatment assignment.
The problem is especially important in policy evaluations where institutions implement several reforms during the same period.
Understanding how concurrent events threaten natural-experiment inference helps determine whether a proposed comparison actually strengthens the study.
How Should Researchers Evaluate Candidate Comparison Groups?
Comparison selection should follow a transparent procedure grounded in the causal design.
Step 1: Define the target estimand.
Specify whose treatment effect is being estimated and which exposure condition constitutes the counterfactual.
Step 2: Reconstruct treatment assignment.
Document why units received or did not receive the intervention, including eligibility rules and implementation timing.
Step 3: Establish an eligible comparison pool.
Identify units with suitable exposure status, outcome definitions, observation periods, and relevant contextual characteristics.
Step 4: Evaluate pre-exposure evidence.
Examine historical outcomes, relevant covariates, treatment-selection mechanisms, and the assumptions required by the intended estimator.
Step 5: Investigate threats to comparability.
Assess spillovers, differential concurrent events, anticipation, missing data, and changes in population composition.
Step 6: Document and test robustness.
Record selection criteria and examine whether conclusions depend substantially on reasonable alternative controls or analytical specifications.
Whenever feasible, researchers should establish comparison-selection criteria before examining treatment-effect estimates.
Choosing the comparison that produces the most favorable result undermines the purpose of constructing a credible counterfactual.
