01 · The Question
When Studies Disagree, Should You Combine Them or Repeat One?
You find several studies addressing the same general question, but their conclusions conflict. One reports a substantial effect, another finds almost nothing, and a third points in the opposite direction. The literature appears unsettled.
You could conduct a systematic review and try to understand the disagreement across the entire evidence base. Or you could replicate a particular study whose result seems especially important or questionable.
The better choice depends on where the uncertainty resides. Is the problem that nobody has adequately compared the existing studies, or is there a particular finding whose reproducibility would materially change how the broader evidence should be interpreted?
03 · What You Need to Know
Different Forms of Disagreement Require Different Research Responses
“The literature is inconsistent” is a description, not yet a diagnosis. Studies can disagree because they examine different populations, use different measures, implement different interventions, follow participants for different periods, apply different analytical decisions, or differ in methodological quality. Sampling variation can also produce apparently conflicting estimates.
Before deciding to replicate anything, establish what kind of inconsistency you are dealing with.
Synthesis problem
Several studies appear to disagree, but the evidence has not been systematically assembled and compared well enough to determine why.
Replication problem
A particular result, procedure, or empirical claim remains consequentially uncertain and reproducing the relevant conditions would provide information the existing studies cannot.
Start With Synthesis When the Disagreement Is Distributed Across the Literature
If ten studies report varying results and nobody has adequately assessed them together, choosing one study for replication may be arbitrary. The most prominent study may not be the source of the uncertainty. It may simply be the study you encountered first.
Systematic review provides a structured way to identify the relevant evidence, compare study characteristics, assess methodological limitations, and determine whether the findings are actually as inconsistent as they initially appear.
Cochrane emphasizes that systematic reviews aim to provide an up-to-date understanding of existing evidence using explicit methods intended to reduce biases arising from ad hoc searching and selection. This is particularly useful when apparently conflicting primary studies have accumulated.
That makes the choice between replication and systematic review partly a question of whether the evidence base itself has already been understood adequately.
Apparent Contradictions May Disappear Once You Compare Like With Like
Imagine one study reports a positive intervention effect while another reports no effect. They seem contradictory until you notice that the first measured outcomes immediately after a four-month intervention, whereas the second measured retention one year later.
Those studies may not be providing competing estimates of precisely the same thing.
Likewise, apparently inconsistent findings may arise from different outcome definitions, comparison groups, participant characteristics, exposures, implementation procedures, or analytical models. Synthesis forces those differences into view before you declare that a finding has “failed to replicate.”
Replication Becomes More Useful When One Finding Carries Disproportionate Weight
Sometimes the disagreement is not distributed evenly. One study may report an unusually large or theoretically important effect while several other studies provide only indirect evidence. Perhaps the study established a widely used claim, influenced practice, or became the foundation for later theoretical work.
If no subsequent study has reproduced the critical conditions closely enough to test that result, a targeted replication may be more informative than another broad synthesis.
The important word is targeted. The replication should address the specific empirical claim generating the uncertainty.
That may require deciding whether to replicate the methodologically strongest study or the most influential one. Those are not necessarily the same target.
Replication Is Especially Valuable When a Finding Has Consequences Beyond the Literature
The informational value of replication can depend on what would happen if the result were wrong.
Suppose one influential study provides much of the empirical basis for a costly educational program, clinical practice, organizational intervention, or policy recommendation. If existing studies do not directly reproduce its central test, uncertainty about that particular finding may deserve priority even when broader evidence exists.
In such cases, researchers may reasonably ask whether they should replicate the study most likely to change practice if its conclusion is wrong. The decision involves consequence as well as statistical disagreement.
Synthesis Is Usually Better for Diagnosing Heterogeneity
If effects vary across many studies, systematic synthesis can help determine whether the variation follows population, context, intervention, measurement, or methodological differences. Meta-analysis may sometimes characterize statistical heterogeneity, while subgroup analysis or meta-regression may explore possible sources when sufficient studies and appropriate data are available.
These analyses have important limitations. Cochrane cautions that investigations of heterogeneity are often difficult when few studies exist and that post hoc subgroup analyses can, at best, generate hypotheses requiring cautious interpretation.
If synthesis reveals a plausible but unresolved explanation, that may then justify a primary study deliberately designed to test it. This is the point at which heterogeneous evidence can justify another primary study.
Replication Is More Compelling When Existing Studies Cannot Recreate the Critical Test
Suppose studies disagree because only one used a particular manipulation, measurement protocol, or analytical procedure central to the original claim. The rest examine related but not equivalent questions.
A meta-analysis of those studies may summarize a broader literature, but it cannot tell you whether that specific result reproduces under the relevant conditions. A direct or closely aligned replication may therefore provide information synthesis cannot.
This distinction matters because studies can be conceptually related without being sufficiently comparable. When the lack of comparable studies prevents meaningful synthesis, targeted new primary evidence may become more valuable.
Do Not Replicate Merely Because One Result Is an Outlier
An extreme estimate may attract attention, but an outlier is not automatically a failed or suspicious study. Sampling variation, population differences, implementation, analytical choices, or genuine effect modification can produce unusual results.
Before choosing the outlier for replication, examine whether there is a plausible reason it differs. If the study is influential and the explanation remains unresolved, replication may be justified. If its difference is readily explained by design or context, repeating it may answer a question you already understand.
Meta-Analysis Is Not Always the Correct Form of Synthesis
Synthesis does not mean that every inconsistent result should be averaged. Cochrane cautions that when study results vary considerably, especially in the direction of effect, quoting an average effect may be misleading.
A systematic review can still compare and appraise the studies without producing a pooled estimate. In fact, refusing to calculate an attractive but scientifically incoherent average is sometimes the more informative analytical decision.
Whether meta-analysis can answer the question better than another primary study therefore depends on comparability as well as the amount of evidence available.
The Strongest Sequence May Be Synthesis, Replication, Then Resynthesis
There is no methodological rule requiring you to choose synthesis forever or replication forever.
A cumulative research program can begin by synthesizing the evidence to locate the disagreement. A replication can then test the critical unresolved finding. Once that study is complete, its result becomes part of the evidence base and can be interpreted alongside the earlier studies.
This sequence avoids treating replication as an isolated verdict on an original paper. A replication result is another piece of evidence, and its meaning depends partly on how it fits with the accumulated research.
Watch Out
Do not select a replication target simply because its result conflicts with your expectations or preferred theory. Define why resolving uncertainty about that particular finding would materially improve interpretation of the evidence before collecting new data.
07 · A Quick Checklist
Before Choosing Replication Over Synthesis
When existing studies disagree, check:
Determine whether the apparently conflicting studies actually address sufficiently similar research questions.
Search for a current systematic review before assuming the inconsistency remains unresolved.
Compare populations, procedures, outcomes, measurements, analytical choices, and methodological limitations across studies.
Identify whether disagreement is distributed across the evidence base or concentrated around a particular result.
Specify what a replication of the proposed target would tell you that synthesis cannot.
Consider the theoretical, methodological, policy, or practical consequences if the target finding does not reproduce.
Design the replication closely enough to address the empirical claim responsible for the uncertainty.
Plan to interpret the replication as part of the accumulated evidence rather than as a final verdict on the earlier literature.
