Will the Study Distinguish Between Competing Explanations?

Will different explanations actually make different predictions in your study?

Researchers often say that a study will “test” an explanation. But imagine that the predicted result appears and several competing explanations predict the same thing. What exactly has been tested?

Finding evidence consistent with an explanation is weaker than showing that the evidence distinguishes it from credible alternatives. A result can support the prediction you wrote down while remaining equally compatible with another mechanism, theory, or causal account.

Before collecting data, the crucial question is therefore not merely whether your preferred explanation predicts an outcome. Ask whether the competing explanations predict sufficiently different outcomes for the study to tell them apart.

A discriminating study creates different evidential consequences for different explanations

The strongest evidence is often evidence that alternatives struggle to explain

Confirmation is easier than discrimination

Suppose your theory predicts that students receiving immediate feedback will perform better than students receiving delayed feedback. You conduct the study and find the predicted difference.

That result is consistent with your theory.

But perhaps an alternative explanation predicts the same difference because immediate feedback also increases time on task. Another explanation might attribute the difference to increased motivation. A third might argue that the interface itself makes the immediate-feedback condition easier to use.

If those alternatives all predict higher performance in the same condition, the observed group difference does not identify which mechanism produced it.

The distinction is subtle but fundamental: evidence can agree with an explanation without discriminating it from alternatives.

Start with alternatives, not only your preferred hypothesis

One influential formulation of this idea is John Platt's account of strong inference. Platt argued for developing alternative hypotheses and designing experiments with possible outcomes capable, as nearly as possible, of excluding one or more of them.

The underlying logic remains useful even when research cannot produce a single decisive experiment. Instead of asking only “What does my theory predict?”, ask:

What else could plausibly produce the same observation?

Then determine whether those alternatives make different predictions somewhere else.

This does not require inventing every logically possible explanation. The useful comparison is among credible alternatives given theory, prior evidence, the design, and domain knowledge.

Write competing predictions before observing the data

A practical way to strengthen a study is to construct a prediction table during planning.

The exact predictions will differ by study. The point of the table is to expose cases in which apparently different explanations are observationally indistinguishable under the planned design.

If every row looks the same across explanations, the experiment cannot discriminate among them no matter how large the sample becomes.

A null hypothesis is not always the most informative competitor

Many studies are organized around a substantive hypothesis versus a null hypothesis of no effect or no difference. That may be appropriate for a particular inferential question, but the scientific competitor is often not “nothing happens.”

Two theories may both predict an effect but disagree about when it should occur, how large it should be, which variable mediates it, how it changes across conditions, or what pattern should appear across outcomes.

Testing only whether an effect differs from zero can therefore leave the substantive competition unresolved.

These comparisons can overlap, but they should not automatically be treated as equivalent.

The explanations must disagree about something observable

Two explanations cannot be distinguished empirically by a particular study if they imply exactly the same observations under every condition that study measures.

You then need a new source of leverage.

Perhaps the explanations disagree about timing. Measure the outcome longitudinally. Perhaps they differ in their predictions for a boundary condition. Introduce that condition. Perhaps one mechanism predicts a response to an intervention that the other does not. Manipulate the mechanism if the design and ethics permit it.

The appropriate design follows from the point at which the explanations diverge.

Discriminating evidence also needs adequate measurement and precision

Different theoretical predictions are not enough if the study cannot observe their differences reliably.

Suppose Explanation A predicts an effect of roughly 2 units and Explanation B predicts roughly 2.2 units. Technically, the predictions differ. But if the planned measurements are noisy and the expected uncertainty around the estimate is several units wide, the study is unlikely to distinguish them.

Conversely, explanations may predict qualitatively different patterns that are comparatively easy to separate.

This is why theoretical discrimination and statistical design must be considered together. The study needs both a prediction gap and sufficient sensitivity to observe it.

Alternative explanations can enter through the design itself

Sometimes the competing explanation is not another formal theory. It is a plausible feature of the study.

Selection bias, confounding, differential attrition, expectancy effects, measurement artifacts, history, or unequal exposure can provide alternative accounts of an observed pattern. Good design attempts to remove, balance, measure, or otherwise address such alternatives.

This is a different but related sense of “competing explanation.” The first concerns theoretical alternatives that make different predictions. The second concerns threats to inference that could produce the same observed result.

Both matter because the evidential question is ultimately comparative: why should this result increase confidence in the proposed explanation rather than its credible rivals?

One study rarely eliminates every alternative

The language of a single “crucial experiment” can be attractive, but many real research problems are not resolved by one decisive test. Theories may be flexible, measurements imperfect, assumptions uncertain, and auxiliary hypotheses necessary to connect abstract explanations with observable predictions.

Failure to distinguish two explanations in one study therefore does not imply that they are permanently indistinguishable. It may indicate that the chosen design lacked the relevant contrast.

Research often progresses through sequences of increasingly discriminating studies rather than one final experiment.

Designing a study so two explanations predict different results

The important change is not merely methodological sophistication. The redesigned study asks the explanations to disagree.

That is the core of discriminating research design.

Why supporting a prediction may not distinguish an explanation

Design the study around where the explanations disagree

Do not begin by asking how to collect more evidence for your favored explanation. Begin by asking what evidence would look different if another credible explanation were true.

This planning process also exposes an important limitation. Sometimes competing explanations are too underspecified to generate distinct predictions. In that case, the problem may lie in the theoretical formulation rather than the statistical analysis.

Ask what each explanation predicts before deciding what to measure. Then ask whether the expected evidence will narrow uncertainty between those explanations enough to matter.

If you cannot identify any observation that would change their relative credibility, collecting another dataset may not resolve the disagreement.

Before testing an explanation, make sure its competitors can lose

Questions about competing explanations and discriminating evidence

Make competing explanations predict different evidence

How to Cite This Guide

This guide is intended to be read, shared, and used in research, teaching, and academic work. If you draw on its ideas, explanations, or other content, please acknowledge the source by citing the guide. Doing so gives appropriate credit and helps your readers locate the original resource.