Should You Replicate the Study Most Likely to Change Practice if Wrong?

Should the Consequences of Being Wrong Determine What We Replicate?

Some research findings remain mostly within scholarly debate. Others influence what teachers teach, clinicians prescribe, organizations implement, governments fund, or institutions recommend.

That difference matters when replication resources are limited. If a study has materially shaped practice and its central finding is wrong, exaggerated, or highly context dependent, the consequences may extend far beyond one incorrect paper.

Does that make it a priority for replication? Often it strengthens the case considerably, but practical influence alone is not enough. You still need to ask how uncertain the finding is, what evidence has accumulated since publication, and whether a replication could actually reduce the uncertainty that matters.

The Consequences of Error Should Matter, but They Are Not the Only Criterion

Replication Priority Depends on Both Uncertainty and Consequence

Replication is not merely repetition. Nosek and Errington characterize replication in terms of whether possible outcomes would provide diagnostic evidence about a prior claim. Under this view, results consistent with the earlier claim should increase confidence in it, while inconsistent results should decrease confidence or reveal limits to its generalizability.

That makes replication particularly relevant when confidence in a claim matters for consequential decisions.

A study can score highly on one dimension without scoring highly on the other. A theoretically fascinating result may have little immediate practical consequence. A relatively unglamorous empirical claim may quietly determine how thousands of people are taught, treated, assessed, or managed.

Ask What Depends on the Finding

Before treating a study as practice changing, trace its actual influence. Is the finding incorporated into guidelines, interventions, curricula, institutional procedures, professional recommendations, commercial products, or policy arguments? Do later researchers merely cite it, or do people actually make decisions because of it?

This distinction matters because citation counts are imperfect indicators of practical consequence. A highly cited paper can have little effect outside scholarship, while a less cited study may influence a widely implemented procedure.

The relevant question is not simply “How famous is this paper?” but “What decisions would look different if the central finding were substantially weaker, stronger, more conditional, or absent?”

The Value of Replication Rises When Important Decisions Rest on Limited Evidence

Imagine an intervention adopted widely after one compelling study. If the original result has little independent confirmation, substantial practical commitment may rest on a narrow empirical foundation.

That creates a potentially valuable replication opportunity. Evidence supporting the original claim could strengthen confidence in continuing the practice. Evidence inconsistent with it could prompt reassessment, further investigation, or recognition that the effect depends on conditions not previously understood.

Nature Human Behaviour has explicitly argued that replication of highly influential research can constitute a significant scientific contribution. More broadly, replication research can strengthen the scientific record by testing whether findings used as foundations for further work are robust.

But “Practice Changing” Does Not Automatically Mean “Replicate Immediately”

Suppose an influential study has subsequently been tested in several rigorous independent investigations and supported by a substantial evidence synthesis. Its original finding may remain consequential, but uncertainty around it could now be relatively low.

Meanwhile, another finding with somewhat less influence may underpin an important practice while resting almost entirely on one small study.

If resources permit only one replication, the second case could offer greater information value. Consequence tells you how much being wrong matters. Existing evidence tells you how plausible and unresolved that possibility remains.

Replication Should Address the Claim That Drives the Decision

A paper may contain dozens of outcomes and analyses. Only some may have influenced practice.

If an intervention was adopted because a study reported improved learning achievement, reproducing a secondary finding about participant satisfaction does little to resolve the decision-relevant uncertainty. The replication should target the empirical claim that actually supports the practice.

This may require identifying the relevant population, intervention, comparator, outcome, follow-up period, and implementation conditions rather than attempting to reproduce every feature of the publication.

The Cost of a False Positive and a False Negative May Differ

Consequences can run in both directions.

If an ineffective intervention is mistakenly believed to work, institutions may waste resources or expose people to unnecessary burdens. But if an effective intervention is mistakenly dismissed, people may lose access to something beneficial.

Replication planning should therefore avoid treating the only interesting outcome as “the original study was wrong.” A rigorous replication is informative because different plausible outcomes update confidence in the claim, not because one outcome makes a better headline.

Practice May Have Moved Beyond the Original Study

A direct reproduction of an old study is not always the most decision-relevant test. Technology, implementation, populations, background conditions, or standard practice may have changed substantially.

In those situations, you need to distinguish reproducibility of the original finding from applicability to current practice. A close replication may tell you whether the original claim reproduces under similar conditions. A more deliberately varied replication may test whether the effect generalizes to the conditions in which decisions are now being made.

Nosek and Errington emphasize that replication is ultimately about whether new evidence is diagnostic of an existing claim, rather than mere duplication of procedures.

Independent Evidence Matters More Than Another Result From the Same Research Ecosystem

When a finding influences consequential decisions, independence can add value. Evidence from another team, population, institution, dataset, or implementation can test whether the original result depends on features that were not obvious from the first study.

This does not mean that every difference improves a replication. Changes must be chosen so that the result remains interpretable relative to the original claim. If too many consequential conditions change simultaneously, a divergent result may be difficult to explain.

Sometimes Synthesis Should Precede Replication

A practice may appear to rest on one famous study even though numerous related investigations have accumulated. Before committing to a new replication, determine what evidence already exists.

If the literature has never been adequately synthesized, systematic review may need to precede replication. The synthesis could reveal that the influential result has already been independently supported, that later findings substantially qualify it, or that one particular uncertainty remains unresolved.

This also prevents confusing the most visible study with the entire evidence base.

Compare Consequential Targets, Not Just Individual Papers

If several studies could be replicated, compare them explicitly. Consider how much each claim influences decisions, how strong the existing evidence is, how much uncertainty remains, whether an informative replication is feasible, and how different outcomes would affect interpretation.

This extends the question of whether to replicate the strongest existing study or the most influential one. Practical consequence provides another dimension that may change which target deserves priority.

When Practical Consequences Raise the Value of Replication

Common Mistakes When Prioritizing Consequential Studies

Prioritize the Intersection of Consequence and Uncertainty

If you are choosing among replication targets, ask two questions together: how much does this finding matter, and how uncertain is it?

A consequential but already well-established claim may not be the highest priority. Neither is a highly uncertain but trivial claim. Replication becomes especially compelling when both dimensions are substantial.

The aim is not to police old papers. It is to direct verification effort toward claims where better evidence has the greatest potential to improve both knowledge and consequential decisions.

Before Prioritizing a Practice-Changing Study for Replication

Questions About Replicating Studies That Influence Practice

Replicate Where Being Wrong Would Matter and Uncertainty Still Remains

How to Cite This Guide

This guide is intended to be read, shared, and used in research, teaching, and academic work. If you draw on its ideas, explanations, or other content, please acknowledge the source by citing the guide. Doing so gives appropriate credit and helps your readers locate the original resource.