How Do You Design a Causal Study to Address the Strongest Alternative Explanation?

What If Your Results Have a Plausible Explanation Other Than the Intervention?

You want to investigate whether AI-supported formative feedback improves university students' academic writing. Students using the system obtain higher final writing scores than students receiving conventional feedback.

However, the intervention group had stronger writing skills before the semester began. Its students may also have been more motivated to revise their work.

You could acknowledge these differences as limitations after completing the analysis. But would it be better to design the study specifically to address them before collecting outcomes?

Suppose prior achievement is the most plausible alternative explanation. Should you randomly assign students, match participants, measure baseline performance, or change the comparison group?

The answer depends on how the alternative explanation could generate the observed result. A strong causal design does not merely accumulate methodological safeguards. It addresses the specific processes that could make the proposed causal conclusion wrong.

Identify the Strongest Rival Explanation Before Selecting the Design

Identifying and Addressing Rival Causal Explanations

What Is an Alternative Explanation in Causal Research?

An alternative explanation is a plausible account of an observed result that differs from the causal interpretation the researcher intends to establish.

For example, an association between AI feedback use and writing performance might arise because feedback improves learning. Alternatively, students with stronger academic motivation may be more likely to use the technology and achieve higher scores.

The second explanation does not require AI feedback to produce the observed association.

Alternative explanations may involve confounding, selection bias, measurement differences, historical events, maturation, regression to the mean, or other mechanisms.

They should be distinguished from mere speculation. A consequential rival explanation must have a plausible relationship to the actual study design and outcome.

Why Should Alternative Explanations Be Considered Before Data Collection?

Many threats to causal inference cannot be repaired adequately after data collection.

If intervention and instructor identity are perfectly confounded, a regression model may be unable to distinguish their separate contributions.

If an important baseline characteristic was never measured, later adjustment may be impossible.

If the comparison group experienced a different assessment procedure, outcome differences may reflect measurement rather than treatment.

Considering these threats prospectively allows researchers to revise assignment procedures, recruitment, measurement, and comparison conditions.

Shadish, Cook, and Campbell (2002) provide a foundational treatment of how experimental and quasi-experimental designs address threats to causal inference.

How Do You Identify the Strongest Alternative Explanation?

Begin with the proposed causal relationship and ask what other process could produce the same result.

For example, if students receiving AI feedback achieve higher scores, consider how they entered the intervention condition, what else differed between groups, and whether the outcome was assessed comparably.

The strongest alternative explanation is not necessarily the one involving the largest number of variables. It is the explanation most capable of accounting for the result under the actual study conditions.

Its plausibility should be evaluated through substantive theory, previous evidence, knowledge of the institutional setting, and the mechanisms generating exposure and outcomes.

Researchers should distinguish plausible threats from hypothetical possibilities unsupported by the research context.

What Are the Most Important Types of Alternative Explanations?

These responses are possibilities rather than universal remedies. Their adequacy depends on the causal question and the assumptions of the selected design.

How Does the Counterfactual Clarify the Problem?

A causal effect compares outcomes under alternative intervention conditions.

For a binary treatment, an average treatment effect may be expressed as:

The central difficulty is that both potential outcomes cannot ordinarily be observed for the same individual under the same circumstances.

Alternative explanations challenge whether the observed comparison group provides credible information about the missing counterfactual outcomes.

For example, students who voluntarily avoid AI feedback may not represent what would have happened to AI users had they not received the intervention.

Hernán and Robins (2020) explain why causal identification requires assumptions connecting observed outcomes with counterfactual quantities.

Can Directed Acyclic Graphs Help Identify Rival Explanations?

Directed acyclic graphs, or DAGs, can represent hypothesized relationships among treatment, outcome, and other variables.

Suppose prior writing ability influences both AI feedback use and final writing performance. A DAG can represent this common-cause pathway.

The graph may also show revision effort as a consequence of feedback use and a potential mediator of its effect.

These variables require different analytical treatment.

Greenland, Pearl, and Robins (1999) explain how causal diagrams clarify confounding and the consequences of conditioning on variables.

However, drawing a DAG does not establish that the assumed relationships are correct. Its value lies in making the causal argument explicit and open to criticism.

Why Is Randomization Particularly Useful Against Selection-Based Explanations?

When participants choose their own intervention condition, treatment selection may depend on characteristics that influence outcomes.

Random assignment can prevent systematic allocation based on baseline potential outcomes in expectation.

For example, randomly assigning students to AI-supported or conventional feedback reduces the plausibility that baseline motivation alone explains differences between assigned groups.

However, randomization does not guarantee identical realized groups or eliminate missing-data bias, contamination, nonadherence, or measurement problems.

The effect of poor adherence in randomized studies illustrates why assignment and actual treatment receipt must be distinguished.

What If Randomization Is Impossible?

Researchers should identify whether the exposure process offers another credible basis for causal comparison.

A policy eligibility threshold may support regression discontinuity when continuity and other relevant assumptions are defensible.

A staggered institutional rollout may support difference-in-differences when its counterfactual trend assumptions are plausible.

Repeated outcome observations may support interrupted time-series analysis when relevant temporal threats can be addressed.

These designs do not simply compensate for the absence of randomization. Each exploits a particular feature of the exposure-generating process.

Researchers should choose the design because its identifying assumptions fit the setting, not because it carries a quasi-experimental label.

How Do You Design Against Confounding?

Confounding occurs when the observed exposure-outcome relationship is distorted by common causes or other relevant causal structures.

Researchers should identify plausible pre-exposure confounders before selecting an analytical model.

For example, prior achievement may influence both voluntary AI feedback adoption and final writing performance.

Possible responses include randomization, restriction, matching, weighting, or regression adjustment under appropriate assumptions.

However, these methods do not automatically address unmeasured confounding.

The broader issue of whether statistical adjustment establishes causation remains relevant even when measured covariate balance appears satisfactory.

How Do You Design Against Historical Changes?

Historical changes may coincide with an intervention and influence the outcome independently.

For example, a university may introduce AI feedback during the same semester that it revises its writing curriculum.

A simple before-and-after comparison cannot reliably distinguish the two changes.

A suitable concurrent comparison group may help address common historical influences.

Repeated observations and appropriately justified longitudinal designs may provide additional evidence.

However, a comparison group must itself be credible. Different institutions may experience different trends, and the assumption of common counterfactual change requires justification.

How Do You Design Against Regression to the Mean?

Regression to the mean is especially relevant when participants are selected because of unusually high or low observed scores.

Suppose students are assigned an intensive feedback intervention because their initial writing assessments were exceptionally poor.

Some subsequent improvement may occur because extreme observed measurements partly reflect random variation.

Without an appropriate reference, researchers may incorrectly attribute this improvement to the intervention.

Repeated baseline measurements, appropriate comparison groups, and careful analysis of selection procedures may help clarify the problem.

Simply demonstrating statistically significant improvement from pretest to posttest does not eliminate regression to the mean.

Why Does Measurement Design Matter for Alternative Explanations?

Observed outcome differences may arise from inconsistent measurement rather than intervention effects.

For example, writing samples from the AI feedback group may be scored by instructors who know which students received the intervention.

Knowledge of treatment assignment could influence scoring, even unintentionally.

Using common assessment tasks, clearly specified rubrics, trained assessors, and blinded scoring where feasible may reduce this concern.

However, standardized measurement must also represent the intended construct. A perfectly consistent measure of the wrong outcome does not establish the intended educational effect.

What If the Intervention Is Confounded With the Instructor?

Suppose one instructor delivers AI-supported feedback while another delivers conventional feedback.

Any observed difference may reflect the feedback condition, instructor practices, or their interaction.

When there is only one instructor per condition, the two effects may be inseparable.

Possible improvements include assigning multiple instructors or classes across conditions or selecting a design that separates instructor and treatment variation.

Statistical adjustment cannot reliably distinguish factors that are perfectly confounded in the available data.

Researchers should address such structural limitations during design rather than describe them as minor statistical inconveniences.

Should Researchers Always Address the Most Likely Alternative Explanation First?

Prioritizing the strongest alternative explanation is often useful, but it should not mean ignoring other serious threats.

For example, randomization may address baseline selection while unequal outcome measurement introduces a different source of bias.

Similarly, a credible comparison group may address historical trends but not selective attrition.

Researchers should identify the principal threats, assess their potential consequences, and select complementary safeguards where necessary.

The aim is to address the most consequential threats systematically rather than accumulate procedures without understanding their purpose.

What Is a Falsification Test?

A falsification test examines a prediction that should hold, or should not hold, if the proposed causal explanation and assumptions are credible.

For example, a negative-control outcome that the intervention cannot plausibly affect may help reveal residual confounding or other bias.

Researchers may also examine whether an apparent intervention effect appears before the intervention begins.

Such findings can challenge the causal interpretation.

However, passing a falsification test does not prove that the identifying assumptions are correct. The test may be insensitive to particular biases, and its usefulness depends on the validity of the negative-control or placebo rationale.

Lipsitch, Tchetgen Tchetgen, and Cohen (2010) discuss negative controls as tools for detecting confounding and bias in observational research.

Can Sensitivity Analysis Address Alternative Explanations?

Sensitivity analysis examines how conclusions depend on assumptions, modeling decisions, or plausible sources of bias.

For example, researchers may investigate how strong unmeasured confounding would need to be to explain an observed association.

They may also examine plausible missing-data mechanisms or alternative definitions of the intervention.

However, sensitivity analysis does not eliminate the bias being investigated.

Its contribution is to show how much the conclusion depends on assumptions that may not be directly verifiable.

Researchers should select sensitivity analyses that correspond to credible threats rather than generate numerous alternative models without a substantive rationale.

What If Several Alternative Explanations Remain Plausible?

Researchers should distinguish threats that can be addressed through additional design features from those that remain unresolved.

For example, a study may address measured prior achievement but still lack information about motivation and informal tutoring.

The resulting adjusted association may be informative without establishing the intervention's causal effect.

Researchers may need to revise the target comparison, seek another identification strategy, or limit the interpretation to association.

A causal claim should not become stronger merely because the researcher has acknowledged several limitations in the discussion.

How Should Alternative Explanations Be Reported?

The methodology should specify the target causal contrast and explain why the selected design addresses the most consequential alternative explanations.

Researchers should identify relevant assignment mechanisms, baseline characteristics, contextual differences, measurement procedures, and analytical assumptions.

Where falsification tests or sensitivity analyses are used, their purpose and limitations should be reported.

The discussion should distinguish threats addressed by design from threats that remain plausible.

Rather than stating generically that confounding may exist, researchers should explain which confounding mechanisms are credible and how they might affect interpretation.

Designing an AI Feedback Study Around Prior Achievement as the Main Rival Explanation

Suppose the randomized comparison produces a smaller positive estimate than the original observational association.

That difference could indicate that baseline selection contributed to the earlier result, although differences in interventions, populations, and precision must also be considered.

The revised design provides stronger evidence about the effect of assignment because it directly addresses the principal selection-based explanation.

The improvement in causal credibility comes from the assignment mechanism and supporting procedures, not merely from obtaining a statistically significant result.

Common Mistakes When Addressing Alternative Causal Explanations

How to Select Design Features That Address Your Strongest Rival Explanation

Begin by writing down the causal conclusion you hope to support. Then identify the most plausible way that conclusion could be wrong.

A defensible design should make it difficult for the strongest rival explanation to account for the result while remaining transparent about other plausible threats.

Check Whether Your Study Addresses Its Strongest Alternative Explanation

Frequently Asked Questions About Alternative Explanations in Causal Research

Design the Study Around What Could Make Your Causal Conclusion Wrong

How to Cite This Guide

This guide is intended to be read, shared, and used in research, teaching, and academic work. If you draw on its ideas, explanations, or other content, please acknowledge the source by citing the guide. Doing so gives appropriate credit and helps your readers locate the original resource.