03 · What You Need to Know
Why Preexisting Group Differences Can Distort Causal Comparisons
Selection Differences Begin Before the Outcome Is Measured
Selection differences arise when the processes determining who receives an intervention, enters a comparison group, or participates in a particular condition produce groups with different characteristics.
These differences may be obvious. For example, one class may have a higher mean pretest score than another. They may also be less visible, involving prior experience, family resources, health status, motivation, institutional support, or other factors not recorded in the dataset.
Consider a study comparing students who voluntarily attend an academic enrichment program with students who do not attend. Participation may reflect motivation, available time, parental encouragement, or concern about academic performance.
Those same characteristics may affect subsequent achievement. Consequently, the final difference between participants and nonparticipants may combine the program's effect with differences arising from the selection process.
The problem is not simply that the groups have different characteristics. It is that some of those characteristics may influence the outcome independently of the intervention.
Why Group Comparability Matters for Causal Inference
A causal effect compares what would happen under different exposure or intervention conditions. For an individual participant, researchers generally cannot observe both potential outcomes under identical circumstances.
Instead, they use outcomes from different participants or observations to estimate the missing counterfactual.
This comparison requires assumptions about exchangeability. Informally, exchangeability means that the groups would have comparable potential outcomes under the same intervention conditions, either unconditionally or after accounting for an appropriate set of baseline characteristics.
For example, if two student groups would have achieved similar outcomes under conventional instruction, a difference observed after one group receives a new intervention may provide evidence about the intervention's effect.
If one group would have performed substantially better even without the intervention, the observed outcome difference cannot be interpreted as the intervention effect without additional assumptions or adjustments.
Exchangeability is not established merely by observing similar group averages. It is a causal condition involving potential outcomes, including outcomes that cannot be observed directly.
How Selection Differences Produce Confounding
Confounding occurs when the observed association between an intervention or exposure and an outcome is distorted by other causes of the outcome.
A common mechanism involves a preexisting characteristic that influences both intervention assignment and subsequent performance.
Suppose prior academic achievement influences whether students are selected for an advanced instructional program. Prior achievement also predicts later examination scores.
In that situation, prior achievement is a potential confounder of the relationship between program participation and examination performance.
The apparent program effect may partly reflect the participants' academic preparation.
Baseline difference
An observed difference in a characteristic measured before the intervention, such as prior achievement or age.
Confounding
Distortion of a causal comparison because other causes of the outcome are associated with the intervention or exposure.
Selection bias
A broader family of biases arising from selection mechanisms, including certain processes of participation, inclusion, retention, or conditioning that distort an association.
These concepts overlap but are not identical. Preexisting differences in prognostic factors can create confounding through nonrandom assignment. Selection bias is also used in a more specific causal-inference sense for distortions introduced by conditioning on participation or other selection variables.
Precise terminology matters because different mechanisms require different safeguards.
Selection Differences Can Arise Through Several Assignment Processes
Nonrandom group assignment is common in educational, organizational, and health research. It is not inherently inappropriate, but researchers must understand the mechanisms generating the groups.
| Assignment process |
Possible selection difference |
Implication for the comparison |
| Voluntary participation |
Participants may differ in motivation, resources, or perceived need. |
Outcomes may reflect self-selection as well as the intervention. |
| Instructor or professional referral |
Individuals may be assigned according to prior performance or assessed need. |
Prognostic characteristics may differ systematically between groups. |
| Existing classes or institutions |
Groups may differ in socioeconomic context, instruction, or prior achievement. |
Group differences may be confounded with institutional or classroom characteristics. |
| Eligibility thresholds |
Participants on opposite sides of a threshold may differ in the assignment variable. |
Simple group comparisons may be misleading, although suitable threshold-based designs can support causal inference. |
| Administrative allocation |
Placement may depend on schedules, capacity, location, or institutional priorities. |
The validity of the comparison depends on whether these factors also influence outcomes. |
Researchers should document the actual assignment process rather than merely labeling the study experimental, quasi-experimental, or observational.
Can Random Assignment Prevent Selection Differences?
Proper random assignment makes intervention allocation independent of participants' baseline potential outcomes in expectation. This is a major advantage of randomized experiments.
Randomization can balance both measured and unmeasured prognostic factors on average across hypothetical repetitions of the assignment process.
However, individual randomized samples may still display baseline imbalances by chance, especially when sample sizes are small.
Such imbalances do not automatically demonstrate a failure of randomization. Researchers should distinguish ordinary chance variation from evidence that allocation was predictable, manipulated, or implemented incorrectly.
Allocation concealment is also important. If recruiters know upcoming assignments, they may consciously or unconsciously influence which participants enter particular groups, undermining the protection offered by a randomized sequence.
Randomization does not prevent every subsequent selection problem. Excluding participants after assignment, selectively losing outcome data, or restricting analysis to adherent participants can undermine the comparability created by the original assignment.
Does Having Similar Baseline Scores Mean the Groups Are Equivalent?
No. Similarity on measured baseline variables does not establish similarity on every characteristic relevant to the outcome.
Two groups may have identical mean pretest scores while differing in prior learning opportunities, motivation, instructional support, or other unmeasured causes of performance.
Even measured similarities can conceal important differences in distributions. Groups with the same average score may have different proportions of high- and low-performing students.
Researchers should therefore examine baseline characteristics selected for their causal relevance, not simply demonstrate that several statistical tests produce nonsignificant results.
A nonsignificant baseline comparison does not establish equivalence. The p-value is affected by sample size and sampling variability, and it does not answer whether an imbalance is causally consequential.
Why Pretest-Posttest Comparisons Do Not Automatically Solve Selection Problems
Collecting baseline outcomes is often valuable. It allows researchers to describe starting conditions, estimate change, and use appropriate adjustment strategies.
Nevertheless, measuring participants twice does not automatically remove confounding.
Suppose the intervention group begins with higher achievement and improves by ten points, while the comparison group improves by five points.
The larger gain may reflect the intervention, but it may also reflect differences in maturation, learning trajectories, exposure to other instruction, or other factors.
Difference-in-differences analysis can help in some nonrandomized settings, but its causal interpretation requires assumptions, notably that the groups would have followed parallel outcome trends without the intervention. This assumption concerns counterfactual trends, not merely similarity in baseline levels.
When multiple preintervention measurements are available, researchers can examine whether observed trends are consistent with the proposed assumption. Such evidence is informative but does not prove that the assumption holds after intervention.
Can Statistical Adjustment Remove Selection Differences?
Regression adjustment, matching, propensity-score methods, weighting, and stratification can help address confounding from appropriately measured baseline variables.
These approaches work differently, but all require careful attention to the variables included and the assumptions supporting causal interpretation.
For example, propensity-score weighting may create a weighted comparison in which measured baseline covariates are more balanced across treatment groups.
However, this does not guarantee balance on unmeasured causes of treatment and outcome. It also requires adequate overlap in the treatment probabilities across relevant participant characteristics.
If students with particular characteristics almost always receive the intervention, there may be insufficient comparable untreated students to estimate the desired effect for that subgroup.
Researchers should avoid treating a propensity score, regression coefficient, or matched sample as proof that selection bias has disappeared.
Watch Out
Do not automatically adjust for every variable associated with group membership or the outcome. Variables affected by treatment, mediators, and colliders may require different handling. Inappropriate adjustment can introduce bias rather than remove it. Covariate selection should follow a defensible causal rationale.
What Makes a Comparison Group More Credible?
A credible comparison group should approximate what would have happened to the intervention group under the alternative condition relevant to the research question.
This requires more than choosing participants who are convenient or superficially similar.
Researchers should consider whether groups arise from comparable recruitment processes, face similar eligibility conditions, share relevant contextual circumstances, and have overlapping distributions of important prognostic factors.
Where possible, assignment procedures should be established before outcomes are known, and baseline characteristics should be measured before intervention exposure.
In some settings, stronger quasi-experimental strategies such as regression discontinuity, instrumental variables, or carefully justified natural experiments may be appropriate. Each depends on design-specific assumptions and cannot be selected merely because the data are nonrandomized.
The most effective approach is to address threats through research design before relying on statistical adjustment.
How Can Selection Differences Interact With Other Threats?
Selection differences may combine with other internal validity problems rather than operating independently.
For example, students in an advanced class may have different rates of academic development than students in a remedial class. Comparing their gains can therefore involve both initial selection differences and different maturation or developmental trajectories.
Likewise, selecting participants because of unusually low baseline scores may introduce concerns about regression to the mean.
If participants subsequently leave the study at different rates, differential attrition may further alter the groups being compared.
These interactions illustrate why internal validity should be evaluated as a property of the actual causal comparison rather than as a collection of isolated labels.
