03 · What You Need to Know
Planning Comparative Research When Baseline Differences Cannot Be Avoided
Why Should Baseline Differences Be Considered During Design?
Baseline differences may influence the interpretation of observed outcomes, particularly when groups are formed through nonrandom processes.
For example, students who voluntarily enroll in an AI-supported learning course may have different prior achievement, technological experience, or motivation from students enrolled in a conventional course.
If those characteristics also influence later performance, the observed outcome difference may not represent the intervention effect alone.
Addressing this problem during design allows researchers to identify relevant variables, choose appropriate comparison groups, and determine whether the intended contrast is feasible.
Waiting until after outcome analysis can encourage selective adjustment or reliance on variables that were not measured adequately.
Shadish, Cook, and Campbell (2002) emphasize the importance of design features and threats to causal inference in experimental and quasi-experimental research.
What Counts as a Baseline Difference?
A baseline difference is a difference between groups in a characteristic measured or defined before the relevant intervention or exposure period.
Examples include prior academic performance, age, program enrollment, digital literacy, access to learning resources, and previous experience with an educational technology.
However, the timing of measurement matters.
A characteristic measured immediately before outcome assessment is not necessarily a baseline variable if it was already influenced by the intervention.
Researchers should establish when the variable arose relative to exposure and whether it could be a cause or consequence of the treatment.
Misclassifying post-treatment variables as baseline covariates can lead to inappropriate adjustment.
Which Baseline Differences Are Most Important?
The most consequential differences are those that threaten the intended inference.
In a causal comparison, particular attention should be given to variables that plausibly influence both treatment selection and the outcome.
For example, prior writing ability may influence whether students choose an AI-supported feedback course and their subsequent writing performance.
By contrast, a characteristic unrelated to exposure selection or outcome generation may not require adjustment for confounding, although it could still affect precision or generalizability.
Researchers should use substantive knowledge and a defensible causal framework rather than selecting covariates solely because they show statistically significant baseline differences.
The distinction between baseline balance and causal comparability is particularly important in nonrandomized studies.
What Should Be Defined Before Choosing an Adjustment Method?
The researcher should first specify the target comparison or estimand.
For example, the objective might be to estimate the average effect of receiving AI-supported feedback rather than conventional feedback among eligible students.
Alternatively, the question might concern the effect among students who actually received the intervention or among students whose baseline characteristics permit meaningful comparison.
These targets are not identical.
Matching or restricting the sample may change the population to which an estimated effect applies.
Hernán and Robins (2020) emphasize the importance of specifying the causal question and identifying assumptions before estimation.
Researchers should not select an adjustment procedure first and then redefine the research question according to whichever estimate becomes available.
What Design Strategies Can Address Baseline Differences?
Several strategies may be considered, depending on the assignment process, available data, and intended inference.
| Strategy |
Potential contribution |
Important limitation |
| Restriction |
Limits eligibility to a more comparable population. |
May reduce generalizability and does not eliminate unmeasured confounding. |
| Matching |
Improves similarity on selected measured characteristics. |
Requires overlap and cannot guarantee balance on unmeasured factors. |
| Regression adjustment |
Models outcomes conditional on selected covariates. |
Depends on model specification and identification assumptions. |
| Propensity-score weighting |
Reweights observations to support a specified comparison population. |
May produce unstable weights and cannot resolve unmeasured confounding. |
| Baseline-adjusted outcome analysis |
Accounts for relevant pre-intervention outcome information. |
Does not independently establish exchangeability. |
| Difference-in-differences |
Compares outcome changes across groups over time. |
Requires defensible counterfactual trend assumptions. |
| Interrupted time series |
Examines changes in outcome levels or trends around an intervention. |
Remains vulnerable to concurrent events and model assumptions. |
These methods are not interchangeable solutions to the same problem. Each targets particular sources of bias and requires a design appropriate to its assumptions.
When Should Researchers Restrict the Study Population?
Restriction may be useful when important differences can be reduced by defining a narrower eligible population.
For example, researchers might include only students enrolled in comparable writing courses with similar assessment requirements.
This may improve the relevance of the comparison by reducing variation in course structure.
However, restriction may also exclude students whose characteristics are important to the broader research question.
It can therefore change the target population and limit generalizability.
Restriction should be justified by the intended inference rather than used merely to produce groups that appear more similar.
When Is Matching Appropriate?
Matching may be appropriate when observational groups differ in measured characteristics and sufficient overlap exists to identify comparable observations.
For example, students receiving AI-supported feedback may be matched with conventional-feedback students based on prior writing performance, academic program, and relevant technological experience.
Stuart (2010) reviews matching methods for causal inference and their role in constructing more comparable observational samples.
However, matching does not automatically establish causal exchangeability.
Unmeasured motivation, informal learning support, or other relevant characteristics may remain different between groups.
Researchers should assess post-matching covariate balance and describe which observations were excluded.
The resulting inference may apply to a matched population rather than every student originally eligible for the study.
What Is Propensity-Score Adjustment?
A propensity score is the conditional probability of receiving a particular treatment or exposure given measured baseline covariates.
For a binary exposure, it can be expressed as:
Propensity scores may be used for matching, weighting, stratification, or other analytical strategies.
However, their usefulness depends on relevant covariates being measured, adequate overlap, and appropriate implementation.
A propensity-score model with excellent treatment prediction does not necessarily provide adequate confounding control.
Researchers should assess covariate balance and the properties of the resulting comparison rather than judge the method solely by predictive accuracy.
How Should Covariate Balance Be Evaluated?
Covariate balance concerns the similarity of measured baseline characteristic distributions after applying a design or adjustment procedure.
Standardized mean differences are commonly used to assess balance.
For a continuous variable, one common form is:
Austin (2009) discusses balance diagnostics for propensity-score matched samples.
Researchers may also examine distributional differences, variance ratios, and overlap rather than relying exclusively on mean differences.
A threshold such as an absolute standardized difference of 0.1 may be used as a practical diagnostic, but it does not universally establish adequate causal comparability.
Balance on measured covariates does not demonstrate balance on unmeasured confounders.
What Is the Problem of Limited Overlap?
Limited overlap occurs when some participant characteristics are represented primarily or exclusively in one exposure condition.
For example, students with very low digital literacy may almost never choose an AI-supported feedback course.
The data may therefore provide little information about how the alternative feedback conditions compare among those students.
Propensity-score weighting may become unstable when estimated treatment probabilities approach zero or one.
Researchers should examine whether the desired comparison is supported by the observed data.
Possible responses include restricting the target population, reconsidering the estimand, or acknowledging that the intended effect cannot be estimated credibly.
Statistical modeling should not be treated as a substitute for missing comparison information.
Should Researchers Compare Posttest Scores or Change Scores?
The appropriate analysis depends on the design and intended estimand.
Comparing final outcomes with adjustment for baseline performance differs from comparing raw changes between pretest and posttest.
Vickers and Altman (2001) discuss the advantages of analysis of covariance in randomized trials with baseline and follow-up measurements.
However, findings from randomized trials should not be transferred mechanically to nonrandomized comparisons.
In observational research, baseline outcome adjustment may address selected differences but does not automatically eliminate confounding.
Researchers should consider the assignment process, measurement error, baseline-outcome relationships, and statistical assumptions before choosing the analysis.
What Is Regression to the Mean?
Regression to the mean occurs when unusually high or low observed measurements tend to be followed by measurements closer to the population average, partly because of random variation.
For example, students selected for an intervention because they obtained unusually low writing scores may show improvement at follow-up even without an effective intervention.
If the comparison group was selected differently, this phenomenon may complicate interpretation.
Researchers should examine how participants were selected, whether baseline values influenced assignment, and whether outcome measurement contains substantial random variation.
Observed improvement among initially low-performing students should not automatically be attributed to the intervention.
Can Difference-in-Differences Address Baseline Differences?
Difference-in-differences can be useful when researchers observe outcomes before and after an intervention in both exposed and comparison groups.
For a simple two-group, two-period arrangement, the descriptive difference-in-differences contrast is:
The approach may account for certain time-invariant differences and common changes over time.
However, causal interpretation requires appropriate assumptions about counterfactual trends, treatment timing, and other relevant conditions.
Parallel trends is a central assumption in conventional difference-in-differences designs. It concerns how the groups' outcomes would have evolved without treatment, not whether their baseline outcome levels were identical.
Researchers should not assume that subtracting baseline differences automatically removes confounding.
What If Baseline Trends Differ?
When repeated pre-intervention observations are available, researchers can examine whether groups followed similar historical trends.
Substantial differences in pre-intervention trends may undermine a simple difference-in-differences interpretation.
However, visually similar pre-intervention trends do not prove that counterfactual trends would have remained parallel after treatment.
Researchers should consider the substantive processes generating outcomes, relevant policy changes, and the possibility of anticipation effects.
Alternative designs or more limited conclusions may be appropriate when the necessary assumptions are difficult to defend.
How Do Clustered Educational Settings Complicate Adjustment?
Students are often nested within classrooms, departments, or institutions.
These settings may influence both intervention assignment and outcomes.
For example, if one intact class receives AI-supported feedback and another receives conventional feedback, instructor and classroom effects may be confounded with treatment.
Even if hundreds of students are observed, there are still only two independently treated classes.
Student-level regression adjustment cannot necessarily distinguish the treatment effect from the difference between those two classes.
Researchers should consider the number of independent assignment units, clustering, and the feasibility of including additional classes or institutions.
The choice of comparison units should be established before selecting a statistical model.
Should Researchers Adjust for Every Baseline Variable?
No. Adjustment should be informed by the causal structure and intended estimand.
Variables that are common causes of exposure and outcome may need to be addressed.
Other variables may improve precision without being confounders.
However, conditioning on colliders or variables affected by treatment can introduce bias or change the causal effect being estimated.
Researchers should avoid selecting covariates solely because they are statistically significant predictors or differ between groups.
Substantive theory and causal diagrams may help identify an appropriate adjustment set.
Can Sensitivity Analysis Address Remaining Uncertainty?
Sensitivity analysis examines how conclusions change under alternative assumptions, analytical decisions, or plausible sources of bias.
For example, researchers may investigate whether an estimated association changes substantially under alternative covariate specifications or reasonable assumptions about missing data.
In observational causal research, sensitivity analyses may also assess how strong unmeasured confounding would need to be to explain an estimated effect.
However, sensitivity analysis does not eliminate bias or prove that the preferred model is correct.
Its contribution is to make the dependence of conclusions on particular assumptions more transparent.
Researchers should prioritize sensitivity analyses that address credible threats to the specific study rather than produce numerous arbitrary alternative models.
How Should Missing Baseline Data Be Handled?
Missing baseline information may limit adjustment and introduce additional uncertainty.
For example, prior writing scores may be unavailable for students who transferred from another institution.
Excluding those students may change the study population and potentially introduce selection bias.
Appropriate missing-data methods depend on the missingness process, variables available, and analytical objectives.
Multiple imputation may be useful under suitable assumptions, but it does not automatically recover information lost through systematic missingness.
Researchers should document missing-data patterns and consider how they affect the credibility of the comparison.
When Should Researchers Reconsider the Comparison Group?
A comparison group may be unsuitable when important determinants of outcomes differ in ways that cannot be addressed credibly.
For example, a university may compare students receiving a new AI-supported writing program with students in an entirely different academic program using different assessment standards.
Even extensive adjustment may not produce a meaningful treatment contrast.
Researchers should consider whether a more appropriate reference group, additional sites, repeated observations, or another design would better address the research question.
The impossibility of perfect comparability does not justify retaining a comparison that lacks a defensible basis.
When Is a Descriptive Conclusion More Appropriate Than a Causal One?
A descriptive or associational conclusion may be more appropriate when important confounding remains unresolved, overlap is inadequate, or the intervention is inseparable from other group characteristics.
For example, researchers may report that students in the AI-supported class obtained higher final writing scores than those in the conventional class.
They may also report how the difference changes after selected baseline adjustment.
However, unless the design and assumptions support causal identification, they should avoid claiming that the AI intervention produced the difference.
Such restraint does not make the study scientifically worthless. It aligns the conclusion with the evidence actually available.
How Should Unavoidable Baseline Differences Be Reported?
The methodology should describe group formation, baseline measurement, covariate selection, and the rationale for the analytical strategy.
Researchers should report relevant baseline distributions and explain how overlap, clustering, missing data, and contextual differences were considered.
When matching or weighting is used, balance diagnostics and changes to the target population should be described.
When longitudinal methods are used, their identifying assumptions should be stated.
The results should include appropriate uncertainty and distinguish prespecified analyses from exploratory adjustments.
The discussion should identify remaining limitations and explain how they affect the intended inference.
Watch Out
Do not select a statistical adjustment procedure merely because baseline differences are statistically significant. First determine why the groups differ, which variables are relevant to the intended causal contrast, and whether the available data support the assumptions required by the chosen method.
