03 · What You Need to Know
Replication Target Selection Is a Research Decision of Its Own
There are usually far more published claims than researchers could feasibly replicate. Target selection is therefore unavoidable.
Pittelkow and colleagues developed a consensus-based checklist for replication target selection in psychology precisely because replication resources are limited and target choices should be justified systematically and communicated transparently. The broader lesson extends beyond psychology: the rationale for selecting a replication target affects what the replication can contribute to the scientific record.
Methodological strength
How convincingly the study's design, measurement, analysis, transparency, and execution support the inference being made.
Scientific influence
How much the study or claim has shaped subsequent research, theory, practice, policy, funding, or other decisions.
These characteristics can overlap, but they need not. A highly rigorous recent study may not yet be influential. An older landmark study may be extremely influential despite methods that would now attract substantial criticism.
Why Replicate the Strongest Existing Study?
Replicating a methodologically strong study can provide independent evidence for a claim under conditions that are already comparatively well specified and defensible.
This may be valuable when the study provides the strongest empirical basis for an important conclusion, when other studies are methodologically weaker, or when the result has not yet been independently reproduced despite its central position in the evidence base.
A successful replication can strengthen confidence that the result is not peculiar to the original sample or research team. A divergent replication can reveal that even apparently strong evidence depends on conditions that were not previously appreciated.
Methodological strength, however, can also reduce the marginal value of replication in some situations. If a strong study is already accompanied by several rigorous independent replications, repeating it again may add less information than investigating a consequential but poorly verified claim elsewhere.
Why Replicate the Most Influential Study?
Influence changes the stakes of uncertainty.
An influential study may anchor a theory, supply the premise for a large subsequent literature, support an intervention, or affect policy and professional practice. If the central finding proves difficult to reproduce, the consequences can extend far beyond one paper.
An editorial in Nature Human Behaviour argued that when an original study is highly influential, rigorous replication of its methodology can itself be highly valuable. Replication initiatives in several fields have similarly targeted influential findings because confirming or challenging them can materially affect the scientific record.
This is especially relevant when considering whether to replicate a study likely to change practice if its result is wrong. Influence is not merely citation count. Consequences matter.
Do Not Treat Citation Count as a Complete Measure of Influence
A heavily cited paper may be cited because researchers criticize it, use its method, discuss the topic historically, or mention it as background. Citation count therefore provides only a partial indication of how much a particular empirical claim matters.
Look instead at what depends on the finding. Has it generated a research program? Is it repeatedly presented as established evidence? Has it shaped a measurement instrument, intervention, theoretical model, guideline, or policy? Would researchers revise important assumptions if the effect were substantially smaller or absent?
Those questions get closer to the kind of influence relevant to replication priority.
The Strongest Study May Not Be the Study Creating the Uncertainty
Imagine a literature containing one recent, carefully designed experiment and an older landmark experiment that established the phenomenon. The newer study may be methodologically stronger, but perhaps it has already been independently supported. Meanwhile, the older study's distinctive central claim remains embedded throughout textbooks and subsequent theory despite limited direct replication.
Replicating the strongest study merely because it is strongest could miss the larger evidential vulnerability.
The target should follow the uncertainty you need to resolve.
The Most Influential Study May Also Be a Poor Replication Target
Influence alone is not enough either. An influential study may use obsolete technology, an intervention no longer practiced, inaccessible materials, or a historical context that cannot meaningfully be recreated.
Its original claim may also have been superseded by stronger subsequent evidence. If several high-quality studies have already tested the relevant finding, replicating the famous original simply because everyone recognizes its title may offer little marginal information.
Replication selection can otherwise become a popularity contest with a methods section.
Ask What Would Change Under Different Replication Outcomes
A useful way to compare candidate targets is to imagine plausible replication outcomes before choosing one.
If the replication closely supports the original result, what becomes more certain? If the estimate is much smaller, what would change? If the result is inconsistent with the original claim, which theories, practices, or research priorities would need reconsideration?
A target has greater potential information value when reasonable replication outcomes would lead to meaningfully different interpretations or decisions.
This perspective helps explain why replication can sometimes be more valuable than a completely novel study. Its contribution may come from reducing uncertainty around evidence on which considerable intellectual or practical weight already rests.
Consider How Much Replication Evidence Already Exists
Do not select a target from the original study alone. Search for direct replications, conceptual replications, robustness analyses, systematic reviews, meta-analyses, and other relevant follow-up evidence.
A study that appears desperately in need of replication when read in isolation may already have been tested repeatedly. Conversely, an influential claim may have hundreds of citations but surprisingly little independent evidence reproducing its critical procedure.
This is why the earlier decision about whether to replicate a study or synthesize the existing evidence first remains important even after you become interested in replication.
Replication Target Selection Can Be Biased
Researchers may preferentially replicate studies that are easy, inexpensive, famous, controversial, or personally doubtful. Such choices can distort what replication evidence eventually represents.
Kühberger and Schulte-Mecklenbeck have argued that non-random selection of replication targets can influence apparent replication success rates. Pittelkow and colleagues likewise emphasize the importance of well-justified and transparent target selection.
This does not mean every replication target must be randomly chosen. It means the selection rationale should be explicit enough that readers can understand why this study, rather than another plausible candidate, received scarce replication resources.
Methodological Weakness Can Increase or Decrease Replication Value
A weak original study can seem like an obvious replication candidate, but weakness creates an interesting tension.
If the original claim is highly consequential and remains influential, a rigorous replication can be valuable because stronger evidence is needed. If the study is weak, little used, and already superseded by better evidence, replicating it may merely confirm that an unimportant weak study was weak.
Likewise, if the original design is incapable of testing the claimed inference, reproducing the same flaw faithfully may have limited value. A replication may need to preserve the critical test while improving aspects necessary for a credible inference, with those departures clearly distinguished from the original procedure.
Sometimes You Should Replicate a Claim, Not a Paper
A paper may contain several experiments, outcomes, or claims. Its most important claim may depend primarily on one experiment or analytical contrast.
Your replication target can therefore be defined at the level of the empirical claim rather than treating the entire publication as indivisible. This makes the rationale clearer: specify exactly what proposition is being tested and what evidence would count as informative replication.
Watch Out
Do not select a study for replication merely because you suspect it will fail. Target selection based on anticipated outcomes can distort the replication literature. Choose targets according to an explicit rationale concerning uncertainty, importance, feasibility, and information value, then design the replication to provide a fair test.
