03 · What You Need to Know
Designing Research That Supports Defensible Causal Inference
What Does a Causal Interpretation Require?
A causal interpretation concerns what would happen to an outcome under alternative intervention or exposure conditions.
For example, the causal question is not simply whether students who use AI feedback obtain higher scores. It is how their outcomes would differ if they received AI-supported feedback rather than the specified alternative.
Under the potential outcomes framework, an average treatment effect can be represented as:
The fundamental difficulty is that both potential outcomes cannot ordinarily be observed for the same individual under the same circumstances.
Holland (1986) discusses this fundamental problem of causal inference. Research designs attempt to address it by constructing credible comparisons that provide information about the missing counterfactual outcomes.
A causal design is therefore evaluated according to whether its comparison and assumptions support the intended effect, not merely whether its statistical model produces a significant coefficient.
Why Must the Causal Question Be Defined Precisely?
Before selecting a design, researchers should specify the population, intervention or exposure, alternative condition, outcome, and relevant timing.
For example, the effect of assigning students access to AI-supported feedback differs from the effect of actually using the system regularly.
Students assigned access may not use it, while frequent users may differ systematically from occasional users.
Similarly, comparing AI feedback with no feedback addresses a different question from comparing it with conventional teacher feedback.
The target causal effect, often called the estimand, should therefore be defined before the analysis.
Hernán and Robins (2020) emphasize the importance of specifying causal questions and the conditions under which effects can be identified.
Without a clear estimand, researchers may produce a statistically precise answer to a question they did not intend to ask.
Which Design Features Strengthen Causal Interpretation?
| Design feature |
Principal contribution |
Important limitation |
| Well-defined intervention |
Clarifies the causal contrast and potential outcomes. |
Ambiguous treatment versions may complicate interpretation. |
| Temporal ordering |
Establishes that the proposed cause precedes the outcome. |
Does not eliminate confounding or alternative explanations. |
| Random assignment |
Supports exchangeability of assigned groups in expectation. |
Does not guarantee adherence or complete outcome data. |
| Credible comparison condition |
Provides information about the relevant counterfactual. |
Selection and contextual differences may remain. |
| Pre-exposure measurement |
Supports assessment of relevant baseline characteristics and adjustment. |
Measured covariates may not capture every confounder. |
| Consistent outcome measurement |
Reduces bias arising from unequal assessment procedures. |
Measurement may still be unreliable or conceptually incomplete. |
| Appropriate causal analysis |
Connects estimation to the design and identification assumptions. |
Incorrect assumptions can produce biased estimates. |
| Sensitivity analysis |
Examines dependence of conclusions on plausible assumptions. |
Cannot prove that all bias has been eliminated. |
These features are complementary rather than interchangeable. Improving measurement does not remove confounding, and randomization does not compensate automatically for an ill-defined intervention or extensive missing outcomes.
Why Is a Well-Defined Intervention Important?
Causal effects are defined relative to particular interventions or exposure conditions.
Suppose researchers classify students as AI users and nonusers without specifying what AI use means.
Some students may use AI to generate complete drafts, others to identify grammatical errors, and others to obtain formative feedback.
These activities may have different effects on writing performance.
A causal estimate combining them may be difficult to interpret because the exposure does not correspond to one clearly specified intervention strategy.
Researchers should define the intervention's content, timing, duration, delivery, and relevant implementation conditions.
For complex interventions, the effect may appropriately concern the complete intervention package rather than a single component.
Why Is Temporal Order Necessary but Insufficient?
A proposed cause must precede the outcome it is claimed to influence.
For example, AI feedback received after a final writing assessment cannot explain performance on that earlier assessment.
Prospective measurement can therefore strengthen the temporal interpretation of an exposure-outcome relationship.
However, temporal precedence does not establish that the exposure caused the outcome.
Students may choose AI feedback because of preexisting motivation or prior achievement, and those characteristics may influence later performance.
The question of whether temporal order alone establishes causation must therefore be considered separately from confounding and counterfactual comparability.
How Does Randomization Strengthen Causal Inference?
Random assignment allocates eligible units to intervention conditions through a chance mechanism.
When properly implemented, it prevents systematic assignment based on baseline potential outcomes and supports exchangeability in expectation.
For example, students could be randomly assigned to AI-supported or conventional feedback.
Under suitable assumptions, comparing outcomes by assigned condition can estimate the effect of assignment.
However, randomization does not guarantee identical realized groups. Chance imbalances may occur, particularly in small samples.
Randomization also does not eliminate missing-data bias, treatment contamination, measurement problems, or interference between participants.
Its contribution should be understood precisely rather than treated as a universal guarantee of causal validity.
What If Individual Randomization Is Not Feasible?
Researchers may consider cluster randomization when intervention delivery or contamination makes individual assignment impractical.
For example, entire writing classes may be assigned to different feedback conditions because students share instructional activities.
However, outcomes from students within the same class may be correlated.
The design and analysis must account for the number of independent clusters, intraclass correlation, and the assignment structure.
A study involving hundreds of students across only a few randomized classes may provide substantially less independent information than the student count suggests.
Cluster randomization can strengthen causal inference when appropriately designed, but it introduces additional requirements for sample size, implementation, and analysis.
How Does a Comparison Group Strengthen Causal Interpretation?
A comparison group can provide information about what might have happened without the focal intervention.
For example, students receiving conventional feedback during the same semester may provide a more relevant reference than students enrolled several years earlier.
However, a comparison group is not automatically a credible counterfactual.
If students choose their own feedback condition, selection differences may explain part of the outcome contrast.
If one instructor delivers the intervention and another delivers the comparison condition, instructor effects may be inseparable from treatment effects.
Researchers should evaluate whether groups are sufficiently comparable for the intended inference.
The existence of two groups is a design feature, not proof of causal identification.
What Is Exchangeability?
Exchangeability concerns whether observed groups can appropriately represent the alternative potential outcomes relevant to the causal comparison.
In observational research, conditional exchangeability may be expressed as:
The assumption means that, after conditioning on L, exposure status provides no additional information about the potential outcomes.
In nonrandomized studies, this assumption cannot generally be verified directly from observed data.
Researchers should justify it through substantive knowledge of treatment selection, relevant causal relationships, and the available measurements.
Similar baseline averages do not independently establish exchangeability.
Why Does Positivity or Overlap Matter?
Positivity requires that the relevant intervention conditions be possible for the covariate profiles represented in the target population.
For example, if students without reliable internet access never receive AI-supported feedback, the available data may not support estimating the effect of that intervention among those students.
Statistical adjustment cannot create observed treatment comparisons where no relevant overlap exists.
Researchers may need to restrict the target population, revise the estimand, or acknowledge that the desired causal effect is not identifiable from the available evidence.
Overlap should therefore be evaluated during design, particularly in observational studies using matching or weighting.
How Does Measuring Baseline Characteristics Help?
Relevant pre-exposure measurements can support adjustment for confounding, improve precision, and clarify the characteristics of the study population.
For example, prior writing ability may be associated with both voluntary AI feedback use and later writing performance.
Measuring it before exposure may permit appropriate adjustment under a defensible causal model.
However, collecting more baseline variables does not automatically improve causal identification.
Some variables may be irrelevant, while conditioning on colliders or post-treatment variables can introduce bias.
Researchers should select covariates according to substantive causal reasoning rather than statistical significance alone.
How Can Directed Acyclic Graphs Help?
Directed acyclic graphs, commonly called DAGs, represent hypothesized causal relationships among variables.
They can help researchers identify potential confounding pathways, mediators, colliders, and appropriate adjustment strategies.
For example, a DAG might represent prior writing ability as influencing both voluntary AI feedback use and final writing performance.
The graph could also represent revision effort as a possible consequence of feedback use and a contributor to performance.
These variables should not necessarily be treated identically in analysis.
Greenland, Pearl, and Robins (1999) explain how causal diagrams can clarify confounding and variable selection.
However, a DAG represents assumptions about the causal structure. Drawing a graph does not establish that those assumptions are correct.
Why Should Researchers Avoid Adjusting for Every Variable?
Indiscriminate adjustment can change the estimand or introduce bias.
For example, if AI feedback increases revision effort and revision effort improves writing performance, revision effort may mediate part of the intervention effect.
Adjusting for it while estimating the total effect could remove part of the causal pathway of interest.
Conditioning on certain colliders can also create noncausal associations.
Researchers should distinguish baseline confounders from mediators and other post-treatment variables.
The appropriate adjustment set depends on the causal question and assumed causal structure.
What Role Does Outcome Measurement Play?
A credible causal design requires outcome measurements that represent the intended construct and are comparable across conditions.
For example, if one feedback group completes a difficult writing task while another completes an easier task, differences in scores may reflect assessment difficulty rather than feedback effectiveness.
Similarly, assessors who know students' treatment assignments may unintentionally score their work differently.
Standardized assessment procedures, appropriate rater training, and blinded outcome assessment where feasible can reduce some measurement biases.
However, consistency alone does not establish construct validity.
Researchers should evaluate whether the outcome actually represents the educational benefit the intervention is intended to produce.
How Does Missing Data Threaten Causal Interpretation?
Missing outcome data may introduce bias when the probability of observation is related to treatment, participant characteristics, or potential outcomes.
For example, students who struggle with an AI feedback system may be more likely to withdraw before the final assessment.
Analyzing only students with complete outcomes may therefore produce a selected comparison that differs from the original assigned groups.
Appropriate methods depend on the missingness process and the intended estimand.
Researchers should document missing-data patterns, minimize attrition where possible, and consider sensitivity analyses addressing plausible departures from primary assumptions.
Randomization does not automatically eliminate bias introduced by selective missing outcomes.
Why Does Treatment Adherence Matter?
Participants may not receive or use an intervention as intended.
For example, students assigned AI-supported feedback may rarely access the system, while some students assigned conventional feedback may use similar tools independently.
These events can complicate interpretation of the intervention contrast.
An intention-to-treat analysis may estimate the effect of assignment under appropriate conditions, but it does not necessarily estimate the effect of sustained intervention use.
Per-protocol or adherence-based causal effects require additional assumptions and suitable analytical strategies.
Researchers should distinguish intervention assignment, actual exposure, and adherence rather than interpreting them as equivalent.
How Can Quasi-Experimental Designs Strengthen Causal Interpretation?
When randomization is infeasible, certain quasi-experimental designs may exploit structured variation in intervention exposure to support causal inference.
Examples include regression discontinuity, interrupted time series, and difference-in-differences designs.
Each requires its own identifying assumptions.
Regression discontinuity may support causal inference near an assignment threshold when relevant continuity and manipulation assumptions are credible.
Interrupted time series may examine changes in outcome levels or trends around an intervention, but remains vulnerable to concurrent events.
Difference-in-differences requires defensible assumptions about how comparison-group outcomes would have evolved without treatment.
Shadish, Cook, and Campbell (2002) provide a foundational discussion of experimental and quasi-experimental designs for causal inference.
These designs are not interchangeable statistical techniques. Their credibility depends on how intervention exposure was generated and whether the assumptions fit the setting.
Why Is a Simple Before-and-After Comparison Often Insufficient?
Students may improve during a semester because of regular coursework, maturation, practice, or other changes occurring over time.
A one-group pretest-posttest comparison cannot ordinarily distinguish these influences from the effect of the intervention.
Even a large improvement does not resolve the identification problem.
Additional design features, such as appropriate concurrent comparison groups, repeated pre-intervention observations, or credible assignment mechanisms, may strengthen causal interpretation.
However, each feature must address a specific alternative explanation rather than merely increase the amount of data collected.
How Does Interference Between Participants Affect Causal Inference?
Interference occurs when one participant's treatment affects another participant's outcome.
For example, students assigned to AI-supported feedback may share generated suggestions with classmates assigned to conventional feedback.
The comparison group's outcomes may then depend partly on the intervention received by other students.
This complicates causal interpretation under assumptions that treat individual outcomes as depending only on individual treatment status.
Researchers may consider cluster assignment, explicit modeling of spillovers, or intervention strategies defined at the group level.
However, cluster randomization does not necessarily eliminate interference across clusters.
The design should reflect how the intervention operates in the actual educational environment.
Can Replication Strengthen Causal Interpretation?
Replication can increase confidence in a causal interpretation when findings recur under different settings, samples, or designs.
For example, an intervention effect observed in one randomized trial may be investigated in additional institutions or through complementary designs.
However, repeated findings do not automatically eliminate shared biases.
Several studies using the same flawed measurement or selection process may reproduce a misleading association.
Replication is most informative when it examines the same causal proposition while varying conditions that might otherwise explain the result.
The question of replication across different research designs is therefore distinct from simply accumulating statistically significant findings.
What Is the Role of Sensitivity Analysis?
Sensitivity analysis examines whether conclusions depend heavily on particular assumptions, measurements, or analytical decisions.
For example, an observational study may investigate how strong unmeasured confounding would need to be to explain an estimated association.
A trial may examine whether conclusions change under plausible missing-data assumptions.
Researchers may also evaluate alternative outcome definitions or prespecified analytical models.
However, sensitivity analysis does not prove that the preferred assumptions are correct.
Its value is to clarify how robust the interpretation is to credible alternatives.
How Do You Identify the Strongest Alternative Explanation?
Researchers should identify plausible processes that could generate the observed association without the focal causal effect.
For example, prior achievement may influence both voluntary AI use and writing performance.
Alternatively, an instructor may introduce additional revision activities at the same time as the AI system.
The first problem concerns selection and confounding. The second concerns an intervention contrast potentially confounded with another instructional change.
The appropriate design response depends on the specific alternative explanation.
A study becomes more credible when its design directly addresses the most consequential threats rather than merely adding statistical controls after data collection.
Does a Strong Design Guarantee a Correct Causal Conclusion?
No. Every empirical design depends on assumptions and imperfect evidence.
Even a well-conducted randomized trial may have limitations involving intervention implementation, outcome measurement, missingness, interference, and applicability to other populations.
Observational designs may rely on assumptions that cannot be fully tested from the observed data.
Researchers should therefore distinguish the strength of the causal evidence from certainty about the conclusion.
A defensible causal interpretation explains which design features support the claim, which assumptions are necessary, and what uncertainty remains.
How Should Causal Design Features Be Reported?
The methodology should define the target causal contrast, population, intervention, comparator, outcome, and timing.
Researchers should describe assignment or exposure-selection procedures, relevant baseline measurements, missing-data handling, and analytical assumptions.
For nonrandomized studies, the rationale for confounder selection and causal identification should be explicit.
For randomized studies, allocation procedures, adherence, deviations, and outcome ascertainment should be reported.
The discussion should distinguish estimated effects from unsupported mechanistic explanations or broader generalizations.
The objective is to make the causal argument open to evaluation rather than present causation as a property automatically conferred by a design label.
Watch Out
Do not treat statistical significance, temporal precedence, randomization, or covariate adjustment as an independent guarantee of causation. Each addresses particular methodological concerns, and a credible causal interpretation depends on the design as a whole.
