03 · What You Need to Know
Understanding Comparability, Exchangeability, and Valid Group Contrasts
What Does Group Comparability Actually Mean?
Group comparability refers to whether the groups provide an appropriate basis for the comparison required by the research question.
Its meaning depends on the intended inference. A descriptive study may legitimately compare average academic performance between two departments even when their student populations differ substantially. The difference is itself the object of description.
However, attributing that difference to a particular teaching method requires additional evidence. Departments may differ in admission criteria, prior achievement, instructional resources, and assessment practices.
Comparability therefore should not be treated as a single property that groups either possess or lack. It is relative to a specified research question, outcome, population, and inferential purpose.
Researchers should first identify the exact comparison required by the research question before deciding which group differences matter.
Do Groups Need to Be Identical?
No. Groups composed of different individuals cannot generally be identical in every characteristic.
Even randomized groups may exhibit chance differences in age, prior achievement, motivation, or other baseline variables.
What matters for causal inference is whether the comparison can support an appropriate counterfactual interpretation, not whether every observed characteristic has the same numerical distribution.
In observational research, the problem is more difficult because the factors determining group membership may also influence the outcome.
Researchers should focus on substantively relevant differences rather than attempting to equalize every available variable.
What Is Exchangeability?
Exchangeability is a central concept in causal inference. Informally, it means that the groups being compared can serve as appropriate representations of the alternative outcomes relevant to the causal question.
For a binary intervention, conditional exchangeability can be expressed as:
Under this assumption, after conditioning on L, treatment assignment does not provide additional information about the potential outcomes.
This assumption is not established merely by observing similar group means. In nonrandomized research, it requires substantive justification and cannot generally be verified directly from the observed data.
Hernán and Robins (2020) explain exchangeability alongside consistency and positivity as important conditions for identifying causal effects.
How Does Randomization Improve Comparability?
Random assignment makes treatment allocation independent of baseline potential outcomes in expectation when implemented appropriately.
This helps prevent systematic selection into treatment conditions based on known or unknown baseline characteristics.
However, randomization does not guarantee that the realized groups will have identical characteristics in a finite sample.
Chance imbalances may occur, especially in small studies. Furthermore, missing outcomes, treatment switching, nonadherence, and other post-assignment events can complicate the interpretation of particular analyses.
Randomization should therefore be evaluated as an assignment procedure, not judged solely by whether a baseline table shows similar group averages.
Why Is Observational Comparability More Difficult?
In observational studies, participants are not necessarily assigned to exposure conditions through a random mechanism.
For example, students who voluntarily use AI-supported feedback may differ from nonusers in motivation, prior writing ability, digital literacy, or access to learning resources.
These characteristics may also influence writing performance.
The observed difference between users and nonusers may therefore combine any effect of feedback use with differences arising from the selection process.
Researchers should identify plausible confounders using substantive knowledge of how exposure and outcomes are generated.
Simply measuring many baseline characteristics does not guarantee that all important sources of confounding have been addressed.
Which Baseline Differences Matter Most?
The most consequential differences are those that threaten the intended inference.
For a causal comparison, particular attention should be given to pre-exposure variables that are common causes of exposure and outcome, along with other variables relevant to identification and precision.
For example, prior writing ability may influence both students' willingness to use an AI feedback system and their subsequent writing performance.
By contrast, a characteristic unrelated to the exposure-selection process or the outcome may be less consequential for confounding, although it may still matter for other design purposes.
Researchers should not select adjustment variables solely because they differ statistically between groups.
Directed acyclic graphs and substantive causal knowledge can help clarify which variables should be considered.
What Is the Difference Between Baseline Balance and Causal Comparability?
Baseline balance concerns similarity in the distributions of measured characteristics across groups.
Causal comparability concerns whether the groups support the intended counterfactual inference.
These concepts are related but not equivalent.
Groups may appear balanced on measured age, gender, and prior achievement while differing in unmeasured motivation or instructional support.
Conversely, randomized groups may exhibit chance differences in measured characteristics while retaining the inferential benefits of valid random assignment.
Baseline balance is therefore useful diagnostic information, but it is not a definitive test of exchangeability.
Should Researchers Use Significance Tests to Check Baseline Balance?
Statistical significance tests are often used to compare baseline characteristics, but their interpretation requires caution.
A nonsignificant difference does not establish equivalence. It may simply reflect limited precision.
Likewise, a statistically significant difference does not automatically establish consequential confounding.
In randomized trials, testing baseline differences to determine whether randomization worked is generally inappropriate because observed differences arise from the random allocation process, assuming it was implemented correctly.
Altman (1985) and subsequent methodological literature have criticized routine significance testing of baseline characteristics in randomized trials.
Researchers should prioritize substantive relevance, distributions, effect sizes, and the design's assignment mechanism rather than treating p-values as a certificate of comparability.
What Are Standardized Mean Differences?
A standardized mean difference expresses the difference between group means relative to a measure of variability.
For a continuous baseline characteristic, one common form is:
Standardized differences can help assess measured covariate balance without relying on sample-size-dependent significance tests.
Thresholds such as an absolute standardized difference of 0.1 are sometimes used as practical diagnostics, but they are not universal proof of adequate balance.
Austin (2009) discusses balance diagnostics for propensity-score matched samples, including standardized differences.
Researchers should examine the substantive importance of the variables, distributions beyond their means, and the relationship between measured balance and the intended inference.
What Is Common Support or Overlap?
Overlap concerns whether comparable treatment or exposure conditions are represented across relevant combinations of participant characteristics.
In causal inference, positivity requires that each intervention condition under consideration have a positive probability for the relevant covariate profiles in the target population.
Suppose every student with very low digital access is in the conventional feedback group, while every student with extensive digital access is in the AI feedback group.
The data may provide little or no direct information about how the alternative feedback conditions compare among students with those extreme profiles.
Statistical adjustment cannot create genuine observations of unsupported exposure conditions.
Researchers may need to restrict the target population, redefine the comparison, or acknowledge that the intended effect cannot be identified adequately from the available data.
How Do Matching and Weighting Affect Comparability?
Matching and weighting are methods that may improve the comparability of observed covariate distributions in nonrandomized research.
For example, propensity-score matching may select intervention and comparison participants with similar estimated probabilities of receiving the intervention, conditional on measured characteristics.
Weighting may instead construct a weighted comparison population.
However, these methods depend on measured covariates, adequate overlap, appropriate modeling, and assumptions about the exposure-selection process.
They do not automatically eliminate unmeasured confounding.
Researchers should also recognize that matching or weighting may change the population to which the estimated effect applies.
The resulting estimand should be reported explicitly.
Can Statistical Adjustment Make Any Two Groups Comparable?
No. Adjustment cannot compensate for every limitation in the underlying design.
For example, if an institution introduced AI-supported feedback at the same time as a new writing curriculum, the effects of those changes may be difficult to distinguish.
Similarly, if important determinants of intervention use and performance were not measured, regression adjustment may leave substantial residual confounding.
Adjustment can also introduce bias when researchers condition on colliders or variables affected by treatment.
Hernán and Robins (2020) emphasize that causal analysis requires a defensible design and identification assumptions rather than statistical modeling alone.
Researchers should therefore address comparability during design whenever possible, rather than treating analysis as a universal repair mechanism.
Does Comparability Require Equal Group Sizes?
No. Groups can differ in size and still support meaningful comparisons.
For example, a study may include 180 students in one feedback condition and 120 in another.
Unequal sample sizes can affect precision and the efficiency of particular analyses, but they do not automatically invalidate the comparison.
Conversely, two groups containing exactly 100 students each may be seriously confounded.
The relevant considerations include the assignment or selection mechanism, measurement, sample composition, and inferential objective.
How Does Measurement Comparability Matter?
Even groups that are otherwise well matched may be difficult to compare if outcomes are measured differently.
For example, one class may complete a writing assessment scored by trained independent raters, while another receives instructor-assigned grades based on different criteria.
Differences in the resulting scores may reflect measurement procedures rather than student performance.
Researchers should establish whether the outcome represents the same construct under comparable conditions.
For multi-item instruments, measurement invariance may be relevant when comparing latent constructs across groups.
Comparability therefore involves both the characteristics of the groups and the quality of the measurement process.
How Do Contextual Differences Affect Group Comparability?
Educational groups are often embedded within different classrooms, departments, or institutions.
These contexts may influence outcomes through instructor practices, assessment requirements, resources, and institutional policies.
For example, comparing students in two departments may inadvertently compare different curricula as well as different feedback systems.
Researchers should identify contextual differences that may influence the outcome and determine whether the design can separate them from the focal contrast.
When students are nested within classes, clustering and the number of independent groups may also affect statistical analysis.
The choice of units of comparison should therefore be consistent with the intended level of inference.
What If the Groups Are Not Comparable?
The appropriate response depends on why comparability is limited and what the research question requires.
Researchers may revise eligibility criteria, improve recruitment, use appropriate design-based adjustment, restrict analysis to an overlapping population, or change the intended inference.
However, these responses have consequences. Restricting the sample may improve support for a comparison while reducing the population to which findings apply.
If the necessary assumptions cannot be defended, researchers should report the observed difference descriptively rather than attributing it causally to the focal intervention.
The broader problem of designing comparisons around unavoidable baseline differences requires explicit attention to the research estimand and remaining uncertainty.
How Do You Know When Comparability Is Sufficient?
There is no single numerical test that certifies two groups as sufficiently comparable for every research question.
Researchers should evaluate the assignment or selection process, relevant baseline characteristics, common support, measurement consistency, contextual conditions, and plausible unmeasured differences.
For causal inference, the central question is whether the counterfactual comparison is defensible under the study's assumptions.
For descriptive research, the groups need not be exchangeable, but their definitions and measurements should support a meaningful interpretation of the observed contrast.
Comparability is therefore an argument supported by design and evidence, not a box checked after obtaining a nonsignificant baseline test.
Watch Out
Do not claim that groups are comparable merely because their baseline p-values exceed 0.05 or their demographic averages appear similar. Neither observation establishes exchangeability, adequate overlap, or the absence of unmeasured confounding.
