03 · What You Need to Know
Understanding What Statistical Adjustment Can and Cannot Accomplish
What Is Statistical Adjustment?
Statistical adjustment refers to analytical procedures that account for selected measured characteristics when estimating a relationship or comparison.
For example, researchers may compare writing performance between AI feedback users and nonusers while accounting for prior achievement and other relevant baseline variables.
Adjustment may be implemented through regression models, stratification, standardization, matching, propensity-score weighting, or other approaches.
These methods differ in how they construct comparisons and the populations to which their estimates apply.
The common objective in observational causal analysis is often to address differences that could distort the relationship between exposure and outcome.
However, adjustment is not synonymous with causal identification. Its usefulness depends on whether the variables, data, model, and assumptions are appropriate for the intended causal question.
Why Do Observational Studies Need to Consider Confounding?
Observational studies generally do not assign exposure through a randomized process.
Students who voluntarily use AI-supported feedback may differ from nonusers in characteristics that also influence academic performance.
For example, students with stronger motivation may be more likely to seek additional feedback and devote more effort to revising their writing.
If motivation influences both feedback use and performance, part of the observed association may arise from that common cause rather than the feedback intervention.
Confounding is therefore a problem involving the causal structure that generates the data, not merely the presence of unequal baseline averages.
Hernán and Robins (2020) explain why observational causal inference requires identifying assumptions that connect the observed data to the intended counterfactual comparison.
What Is the Difference Between Adjustment and Causal Identification?
Adjustment is an analytical operation. Causal identification concerns whether the intended causal effect can be expressed using the observed data under a set of assumptions.
For example, a regression model may estimate the association between AI feedback use and writing performance conditional on measured covariates.
That coefficient is not automatically the causal effect of AI feedback.
To interpret it causally, the researcher must justify why the adjusted comparison represents the relevant alternative potential outcomes.
Even a technically well-fitted model may estimate a biased association if important confounders are missing or the exposure-selection process is misunderstood.
Researchers should therefore establish the causal question and identification strategy before selecting the statistical procedure.
What Assumptions Are Important for Adjusted Causal Comparisons?
| Assumption or requirement |
Meaning |
Potential problem |
| Well-defined intervention |
The treatment conditions correspond to meaningful causal contrasts. |
Different forms of AI use are combined without specifying the intervention. |
| Temporal ordering |
The proposed exposure precedes the outcome. |
AI use is measured after writing performance has already been assessed. |
| Conditional exchangeability |
Relevant confounding is adequately addressed by the selected covariates. |
Unmeasured motivation influences both exposure and outcome. |
| Positivity |
Relevant exposure alternatives are possible across target covariate profiles. |
Some student groups never receive one feedback condition. |
| Consistency |
Observed outcomes correspond to the relevant potential outcomes under the exposure actually received. |
The treatment label combines incompatible intervention versions. |
| Appropriate measurement and modeling |
Variables and estimation procedures adequately represent the intended relationships. |
Important confounders are measured poorly or modeled incorrectly. |
These conditions are not automatically satisfied by including additional covariates in a regression model.
What Is Conditional Exchangeability?
Conditional exchangeability is the assumption that, after accounting for an appropriate set of pre-exposure covariates, exposure status is independent of the relevant potential outcomes.
For a binary intervention, it may be represented as:
In practical terms, the assumption requires that the adjusted groups provide an appropriate basis for estimating the intended causal contrast.
However, conditional exchangeability cannot generally be verified directly because both potential outcomes are not observed for the same individual.
Researchers must justify the assumption through substantive knowledge, study design, and the plausibility of the measured confounder set.
Good statistical model fit does not establish conditional exchangeability.
Why Is Unmeasured Confounding Such an Important Limitation?
Statistical adjustment generally operates on variables available in the dataset.
If an important common cause of exposure and outcome was not measured, ordinary adjustment cannot directly account for it.
For example, a study may measure prior achievement and digital literacy but omit students' willingness to seek additional academic assistance.
If that characteristic influences both AI feedback adoption and writing improvement, the adjusted association may remain confounded.
Researchers should distinguish the absence of a measured imbalance from evidence that no unmeasured confounding exists.
The former is observable; the latter ordinarily requires assumptions and additional evidence.
Does Including More Covariates Make the Causal Estimate More Reliable?
Not necessarily. Adding variables can improve adjustment when they are relevant confounders, but indiscriminate inclusion may introduce bias or unnecessary variance.
Some variables may be consequences of the exposure rather than baseline causes.
For example, AI feedback may increase revision effort, which subsequently improves writing performance.
If the researcher wants to estimate the total effect of AI feedback, adjusting for revision effort may remove part of the causal pathway.
Other variables may act as colliders, and conditioning on them can introduce associations that distort the intended comparison.
Greenland, Pearl, and Robins (1999) demonstrate how causal diagrams can clarify the consequences of conditioning on different variables.
Covariate selection should follow causal reasoning rather than the number of available columns in a dataset.
What Is Collider Bias?
Collider bias can arise when researchers condition on a variable influenced by two other variables, potentially creating a noncausal association between them.
Suppose participation in an optional AI feedback workshop depends on both student motivation and prior academic difficulty.
Restricting the analysis to workshop participants may create an association between motivation and academic difficulty even if the relationship differs in the wider population.
That induced association may distort an exposure-outcome comparison.
Collider bias illustrates why controlling more variables or restricting the sample does not necessarily improve causal validity.
Researchers should consider the assumed causal relationships before conditioning on variables.
Why Is Adjusting for Mediators Different From Adjusting for Confounders?
A confounder may create a noncausal association between exposure and outcome. A mediator lies on a causal pathway through which the exposure may influence the outcome.
For example, prior writing ability may confound voluntary AI feedback use and final writing performance.
By contrast, increased revision effort may be a consequence of receiving AI feedback and part of how the intervention affects performance.
Adjusting for revision effort may therefore change the estimand from a total effect toward a different causal quantity.
Estimating direct or mediated effects requires additional assumptions and appropriate methods.
Researchers should specify whether the objective concerns the total effect or a particular causal pathway before selecting adjustment variables.
What Is the Difference Between Regression Adjustment and Propensity-Score Methods?
Regression adjustment commonly models the outcome as a function of exposure and selected covariates.
Propensity-score methods model or use the probability of exposure conditional on measured covariates.
For a binary exposure, the propensity score is:
Propensity scores may support matching, weighting, or stratification.
Stuart (2010) reviews matching methods for causal inference and their methodological considerations.
Neither regression nor propensity-score methods automatically eliminate unmeasured confounding.
Their suitability depends on the target estimand, overlap, measurement, model assumptions, and the substantive exposure-selection process.
Does Propensity-Score Matching Make an Observational Study Equivalent to a Randomized Trial?
No. Matching may improve similarity between groups on selected measured covariates, but it does not reproduce the assignment mechanism of a randomized trial.
For example, AI users and nonusers may be matched on prior writing scores, program, and digital literacy.
After matching, the groups may still differ in unmeasured motivation, access to informal support, or other determinants of performance.
Researchers should assess measured covariate balance and recognize that causal interpretation remains conditional on assumptions.
Matching can strengthen an observational comparison without transforming it into an experiment.
What Is the Role of Covariate Balance?
Covariate balance describes the similarity of measured baseline characteristics across the comparison groups, potentially after matching or weighting.
Standardized mean differences and other diagnostics can help assess balance.
Austin (2009) discusses the use of balance diagnostics in propensity-score matched samples.
However, balance on measured characteristics does not establish balance on unmeasured confounders.
Researchers should also evaluate whether important covariate profiles are represented across both exposure conditions.
A visually reassuring balance table is not a substitute for a defensible causal identification argument.
Why Does Positivity Matter for Statistical Adjustment?
Positivity requires that the relevant exposure conditions be possible across covariate profiles in the target population.
Suppose students without reliable internet access never use the AI feedback system.
The dataset may provide no direct comparison of AI use and nonuse among students with those access characteristics.
Regression may still generate predictions for unsupported combinations of characteristics, but those predictions depend on extrapolation.
Propensity-score weighting may also become unstable when exposure probabilities approach zero or one.
Researchers should assess overlap and consider restricting the target population or revising the estimand when necessary.
Can Statistical Adjustment Correct Selection Bias?
Some forms of selection can be addressed using appropriate design and analytical methods under defensible assumptions.
However, selection bias may arise when inclusion in the analyzed sample depends on exposure, outcomes, or related variables.
For example, students who find AI feedback difficult may be more likely to withdraw before the final writing assessment.
Analyzing only students with complete data may produce a selected sample that no longer represents the intended population.
Ordinary baseline regression adjustment does not necessarily resolve this problem.
Researchers should distinguish confounding control from methods addressing missing outcomes or selection into observation.
Why Is Temporal Order Still Important After Adjustment?
Statistical adjustment cannot repair an incoherent temporal sequence.
For example, adjusting for prior achievement does not establish that AI feedback caused an outcome if feedback use was measured only after that outcome occurred.
Researchers should establish when exposure began, when covariates were measured, and when outcomes occurred.
Variables measured after exposure may have different causal roles from genuine baseline confounders.
The broader question of whether temporal order is sufficient for causation illustrates why chronology and confounding control must be evaluated together.
Can Longitudinal Adjustment Introduce Additional Problems?
Yes. Longitudinal studies may involve time-varying confounders affected by earlier exposure.
For example, early AI feedback use may influence students' academic confidence. Confidence may subsequently influence continued AI use and later writing performance.
Conventional regression adjustment for confidence at each time point may be inappropriate for some total-effect questions.
Methods such as inverse probability weighting, the parametric g-formula, or other g-methods may be relevant under appropriate assumptions.
However, these methods require careful measurement of exposure and confounding histories and do not automatically eliminate unmeasured bias.
What Is Model Misspecification?
Model misspecification occurs when the statistical model inadequately represents relevant relationships in the data.
For example, a regression may assume that prior writing ability has a linear relationship with final performance when the relationship is substantially nonlinear.
The model may also omit relevant interactions or use inappropriate functional forms.
Even when the required confounders are measured, an unsuitable model may produce biased estimates.
Flexible modeling and diagnostic procedures can sometimes reduce these concerns, but they do not replace causal identification assumptions.
Researchers should distinguish model fit from the validity of the causal interpretation.
Can Doubly Robust Methods Solve the Problem?
Doubly robust estimators combine information from models of treatment assignment and outcomes.
Under appropriate conditions, certain estimators remain consistent if at least one of the relevant nuisance models is correctly specified.
However, double robustness does not mean protection against every form of bias.
Unmeasured confounding, violations of positivity, inconsistent treatment definitions, and serious measurement problems may remain.
Researchers should understand the assumptions of the particular estimator rather than interpret the term doubly robust as a guarantee of causal validity.
What Is Sensitivity Analysis for Unmeasured Confounding?
Sensitivity analysis investigates how conclusions might change under specified departures from the primary causal assumptions.
For example, researchers may examine how strong an unmeasured confounder would need to be, in relation to exposure and outcome, to explain an observed association.
VanderWeele and Ding (2017) introduce the E-value as one approach to sensitivity analysis for unmeasured confounding on a risk-ratio scale.
However, the E-value is not a universal measure of causal validity and does not account for every type of bias.
Sensitivity analysis should be selected according to plausible threats in the particular study.
Its purpose is to clarify dependence on assumptions, not prove that no alternative explanation exists.
When Can an Adjusted Observational Estimate Support a Causal Interpretation?
An adjusted observational estimate may support a causal interpretation when the intervention contrast is well defined, temporal alignment is appropriate, and the identification assumptions are substantively defensible.
The researcher should explain how exposure was selected, why the chosen covariates are sufficient for the intended adjustment, whether overlap exists, and how measurement and selection biases were addressed.
Some observational settings offer stronger identification opportunities than others, including policy thresholds, staggered interventions, or credible natural experiments.
However, the presence of an advanced statistical estimator does not independently establish that the underlying design supports causation.
The broader research design features that strengthen causal interpretation should be considered together.
How Should Researchers Report Adjusted Observational Findings?
The methodology should identify the target causal question, exposure definition, covariate selection rationale, statistical approach, and relevant assumptions.
Researchers should describe the timing of measurements, missing-data procedures, overlap diagnostics, and sensitivity analyses where appropriate.
The results should distinguish unadjusted and adjusted estimates without implying that the adjusted result is necessarily the true causal effect.
When causal assumptions remain uncertain, the discussion should acknowledge the resulting limitations explicitly.
Researchers should not use the phrase controlled for all confounders unless such a claim is genuinely defensible, which is rarely possible in ordinary observational research.
Watch Out
Do not interpret a statistically significant adjusted coefficient as proof of causation. Adjustment addresses selected measured relationships under assumptions; it does not automatically eliminate unmeasured confounding, selection bias, model misspecification, or inadequate overlap.
