03 · What You Need to Know
Designing Comparisons That Support the Intended Causal Inference
What Is a Comparison Group?
A comparison group consists of individuals or units whose outcomes provide evidence about an alternative condition to the exposure or intervention of interest.
In a randomized experiment, participants may be assigned to an intervention or a specified comparator. In observational research, groups are formed from naturally occurring treatment choices, exposures, or institutional practices.
The comparator may involve no intervention, usual practice, another intervention, or a different level of exposure.
Its scientific purpose is not merely to provide a second set of observations. It is to support a meaningful contrast between clearly defined conditions.
For example, comparing AI-supported feedback with conventional instructor feedback answers a different question from comparing AI-supported feedback with no feedback.
Why Must the Causal Question Come Before Group Selection?
The appropriate comparison depends on the effect researchers intend to estimate, often called the estimand.
Suppose the question is whether offering an AI-supported platform improves achievement. The relevant contrast may be assignment to platform access versus assignment to usual instruction.
If the question instead concerns the effect of actually using the platform regularly, the treatment strategies and causal assumptions differ.
Researchers should specify the population, treatment conditions, outcome, follow-up period, and summary measure before choosing participants.
Otherwise, the available data may determine the comparison retrospectively, encouraging researchers to interpret an accessible contrast as though it answered their original question.
How Is Selection Bias Different From Confounding?
Selection bias and confounding can both distort comparisons, but they describe different causal problems.
Confounding
A noncausal pathway, often through a common cause of exposure and outcome, distorts the exposure-outcome relationship.
Selection bias
Conditioning on inclusion, observation, or another selection process distorts the relationship or quantity being estimated.
For example, motivated students may be more likely to use an optional platform and perform better academically. This creates a potential confounding problem.
Separately, restricting the analysis to students who completed the semester may create selection bias if completion depends on platform use and outcome-related characteristics.
The problems may coexist, but matching students on motivation does not necessarily correct selection introduced by conditioning on completion.
Why Should Groups Share Compatible Eligibility Criteria?
Groups should be drawn from populations for which the specified treatment strategies and comparison are meaningful.
For example, comparing students eligible to receive an AI platform with students at institutions where the platform could never be implemented may introduce substantial contextual differences.
Eligibility should ordinarily be assessed using information available before the treatment strategies begin.
Researchers should avoid applying one set of criteria to intervention participants and another to comparison participants without a substantive justification.
However, identical written eligibility rules do not automatically create comparable groups. The actual mechanisms determining treatment and inclusion still matter.
The implications of eligibility restrictions for selection bias should be evaluated before recruitment or record extraction.
What Is a Common Time Zero?
Time zero is the point at which eligibility, treatment assignment or treatment-strategy classification, and the start of outcome follow-up are aligned.
In a randomized trial, this often corresponds to randomization. In an observational emulation, researchers must define an equivalent starting point.
Suppose platform users are classified only after completing four weeks of use, while nonusers are followed from the first day of the semester.
Users must remain enrolled and observable long enough to satisfy the exposure definition. This creates an opportunity for immortal time bias because early failures may be excluded or allocated differently.
Researchers should establish comparable starting conditions rather than classifying groups using future information without an appropriate design.
Why Can Comparing Users With Nonusers Be Misleading?
Actual users and nonusers may differ in characteristics that influence both technology use and academic outcomes.
Some differences may be measured at baseline, such as prior achievement or connectivity. Others, such as motivation or instructor encouragement, may be incompletely observed.
Furthermore, nonuse can arise for different reasons. A student may decline an available platform, lack access, or discontinue after encountering difficulties.
These are not necessarily equivalent comparison conditions.
Researchers should define the treatment strategies precisely and distinguish baseline treatment choice from postbaseline adherence.
When feasible, random assignment to access or encouragement may provide a more credible estimate of a clearly specified policy effect.
When Is Randomization the Preferred Solution?
Randomization assigns treatment conditions through a chance mechanism rather than allowing participant characteristics to determine assignment.
With adequate implementation, it supports comparability in expectation for baseline causes of the outcome, including unmeasured ones.
Randomization does not guarantee identical groups in a finite sample. It also does not prevent contamination, differential attrition, measurement bias, or nonadherence.
Researchers should therefore protect allocation concealment where relevant, document assignment procedures, and preserve follow-up.
When individual randomization is impractical, cluster randomization or other experimental designs may be appropriate, provided their assumptions and analytical implications are addressed.
What Makes an Observational Comparison Group Credible?
An observational comparison should approximate the treatment decision that would have been made in a well-defined hypothetical randomized trial.
Hernán and Robins describe this approach through target trial emulation, which specifies eligibility, treatment strategies, assignment procedures, time zero, outcomes, follow-up, and analysis.
The comparison should involve people who could plausibly have received either strategy under the research question.
For example, students offered two alternative instructional formats within the same course may provide a more interpretable comparison than students from unrelated institutions with substantially different curricula.
However, shared context does not eliminate unmeasured confounding. The treatment-selection mechanism still requires investigation.
What Is Confounding by Indication?
Confounding by indication occurs when the reasons for receiving an intervention are associated with the outcome.
Although the term is common in clinical research, analogous processes occur in education.
For example, instructors may assign additional AI tutoring to students who are already struggling.
Comparing those students with peers who did not receive tutoring may make the intervention appear less effective because the treated group had poorer expected outcomes before treatment.
Conversely, highly motivated students may volunteer for supplementary learning opportunities, potentially exaggerating apparent benefits.
Researchers should identify why treatment was assigned or chosen and measure relevant pretreatment characteristics where feasible.
Can Matching Create Better Comparison Groups?
Matching selects or weights observations to improve comparability on specified measured characteristics.
For example, students receiving an intervention may be matched with comparison students who have similar prior achievement, academic level, and baseline connectivity.
This can reduce measured differences, but it does not guarantee that unmeasured causes of treatment and outcome are balanced.
Matching can also change the population represented by the estimate when some participants lack suitable counterparts.
Researchers should examine whether matching reduces confounding or introduces new selection problems.
Variables should be selected according to their causal roles rather than their availability or statistical significance alone.
What Is the Role of Propensity Scores?
A propensity score is the probability of receiving a specified treatment conditional on measured pretreatment characteristics.
Propensity scores may be used for matching, weighting, stratification, or adjustment.
Their usefulness depends on whether the measured variables are sufficient to address confounding, whether positivity holds, and whether the analytical implementation is appropriate.
A propensity score model with strong predictive performance does not necessarily provide good causal adjustment.
Researchers should examine covariate balance after adjustment and consider the target estimand.
Even excellent balance on measured variables cannot establish balance on unmeasured confounders.
Why Does Overlap Matter?
Overlap, or positivity, concerns whether individuals with relevant covariate profiles have a positive probability of receiving each treatment strategy being compared.
Suppose students with very limited connectivity never receive the AI platform.
The available data contain no treated students in that subgroup, making its treatment effect difficult to identify without extrapolation.
Matching may discard unmatched students, while weighting may generate extreme weights where treatment probabilities approach zero or one.
Researchers should examine the distribution of relevant characteristics across groups and clarify the population to which the resulting effect applies.
Restricting to an overlap population may be defensible, but it changes the estimand and potentially its applicability.
Should Comparison Groups Come From the Same Institution?
Not necessarily. Selecting participants from the same institution may reduce differences in curriculum, assessment procedures, and organizational context.
However, shared institutional membership does not ensure comparable treatment assignment.
For example, one department may offer the platform primarily to students with lower prior achievement.
Conversely, using different institutions may be appropriate when treatment is assigned at the institutional level, but site-level differences require careful consideration.
The decision should follow the causal question, treatment-assignment mechanism, and available variation.
Researchers should not assume that geographic or institutional proximity is a substitute for causal comparability.
Why Must Outcome Measurement Be Comparable?
Comparison groups should ordinarily have outcome measures that support the same interpretation under comparable assessment conditions.
For example, comparing examination scores from two courses with different content, difficulty, and grading rules may be misleading even when student characteristics appear similar.
Researchers should establish the intended outcome construct, assessment timing, scoring procedures, and relevant measurement equivalence.
Measurement problems are distinct from selection bias but can compound its effects.
Appropriate standardization of measurement across study groups helps preserve the interpretation of the comparison.
Can Posttreatment Selection Undermine an Otherwise Good Comparison?
Yes. A comparison established at baseline may become distorted if researchers later analyze only participants who adhere, complete the intervention, or provide final outcomes.
For example, excluding students who discontinue platform use can condition on characteristics affected by treatment and related to achievement.
This may undermine the original treatment comparison.
Researchers should distinguish treatment discontinuation from missing outcomes and choose methods consistent with the intended estimand.
The consequences of selective loss to follow-up require explicit evaluation.
Can a Negative Control Help Assess Comparison-Group Bias?
Negative control exposures or outcomes may help investigate certain residual biases when their causal assumptions are defensible.
For example, a negative control outcome should not plausibly be affected by the intervention but may share relevant sources of confounding or measurement bias.
An unexpected association with that outcome can raise concerns about the design.
However, negative controls do not automatically identify or eliminate bias, and suitable controls may be difficult to find.
They are complementary diagnostic tools rather than replacements for careful comparison-group construction.
How Should Researchers Evaluate Comparison-Group Quality?
Begin with the causal structure and study protocol, then examine observable evidence.
Useful checks may include eligibility alignment, treatment assignment mechanisms, common time zero, baseline covariate distributions, overlap, outcome ascertainment, and attrition.
Balance diagnostics can reveal measured differences but cannot prove exchangeability.
Researchers should avoid using baseline significance tests as the principal criterion for comparability, particularly in randomized studies where such tests do not establish successful randomization.
A credible comparison rests on design and assumptions, supported by appropriate diagnostics.
How Should Comparison-Group Selection Be Reported?
Reports should explain the target estimand, eligibility criteria, treatment definitions, source population, assignment mechanism, and time zero.
For observational studies, describe the variables used for confounding adjustment, the rationale for their selection, overlap diagnostics, and exclusions created by matching or weighting.
Report the comparison condition and outcome measurement procedures sufficiently for readers to evaluate whether the contrast is meaningful.
Where important differences remain, explain their implications without claiming that statistical adjustment has established equivalence.
Transparent reporting makes the causal assumptions inspectable rather than presenting a balanced table as definitive proof of validity.
