03 · What You Need to Know
How to Tell Genuine Contradiction From Ordinary Variation
First Check Whether the Studies Are Answering the Same Question
Two studies cannot meaningfully contradict one another if they are answering materially different questions.
Suppose one study finds that an intervention improves immediate test performance among university students, while another finds no improvement in long-term retention among primary-school pupils. The findings differ, but they concern different populations and outcomes. Calling them contradictory collapses distinctions that may actually explain the difference.
Before comparing findings, align the relevant elements of the research question:
- population;
- intervention, exposure, phenomenon, or predictor;
- comparison where applicable;
- outcome;
- time point;
- context or setting; and
- the substantive claim being tested.
Only then can you determine whether the evidence genuinely pulls in different directions.
Different Effect Estimates Are Normal
Studies sample different people and operate under different circumstances, so identical results should not be expected.
Cochrane describes variability among studies broadly as heterogeneity. Clinical diversity can arise from differences in participants, interventions, and outcomes, while methodological diversity can arise from differences in study design, outcome measurement, and risk of bias. Statistical heterogeneity refers to effect estimates differing more than would be expected from sampling variation alone.
Some heterogeneity is therefore normal. Contradiction becomes more consequential when differences change the substantive interpretation.
Variation
Studies produce somewhat different estimates but remain compatible with the same broad substantive conclusion.
Contradiction
Studies support materially incompatible conclusions about the same substantive question.
Do Not Compare Statistical Significance Labels
One study reports p <.05. Another reports p =.08. That does not establish contradictory findings.
The studies might estimate almost identical effects but have different sample sizes or precision. One confidence interval may simply be wider than the other.
Instead, compare effect estimates and their uncertainty. Ask whether the results are compatible with similar underlying effects and whether any difference is substantively meaningful.
The same principle applies outside conventional null-hypothesis testing. Compare the actual findings and claims rather than binary labels attached to them.
Direction of Effect Can Signal Important Contradiction
Differences become particularly important when credible studies estimate effects in opposite directions.
For example, some studies might suggest that an intervention improves an outcome while others suggest that it worsens it. Cochrane specifically advises reviewers to take heterogeneity into account when interpreting results, particularly when the direction of effect varies across studies.
Even then, do not stop at “results conflict.” Ask whether population, implementation, dosage, setting, measurement, bias, or another characteristic explains the reversal.
Magnitude Can Contradict Even When Direction Does Not
Contradiction is not restricted to positive versus negative effects.
Suppose one set of rigorous studies indicates a substantial benefit while another indicates an effect so small that it would be practically negligible. All estimates may technically lie on the same side of zero, yet they support different practical interpretations.
Define what counts as a meaningful difference for the question. Otherwise, you risk treating every numerical discrepancy as contradiction or overlooking disagreements that matter simply because their signs match.
Look for Population Differences That Explain the Conflict
Apparent contradiction may reveal effect modification rather than scientific chaos.
An intervention might benefit one population but have little effect in another. A relationship may be stronger at one developmental stage. Institutional resources, baseline risk, prior experience, or socioeconomic conditions may alter an effect.
Cross-reference conflicting findings with the populations represented across studies. If findings cluster systematically by population, the literature may be telling you something more interesting than “results are inconsistent.”
Look for Methodological Explanations
Studies may disagree because their methods create different opportunities for bias or because they operationalize the question differently.
Cochrane notes that methodological diversity, including differences in study design and outcome measurement, can contribute to heterogeneity. If studies with one methodological feature systematically produce larger or smaller effects, the conflict may reflect design differences rather than genuine variation in the phenomenon.
Compare contradictory findings with the methodological structure of the literature. Ask whether study design, sampling, follow-up, missing data, confounding, or analysis tracks the pattern of results.
Measurement Can Manufacture Apparent Contradiction
Two studies may use the same outcome label while measuring different constructs.
For example, “engagement” might refer to self-reported interest in one study and observed platform activity in another. “Achievement” might refer to standardized performance in one paper and self-reported grades in another.
If those measures behave differently, the findings may not genuinely contradict one another. They may reveal that the supposedly common outcome is not common at all.
Outcome mapping is therefore essential before interpreting conflicting results. Examine whether repeatedly studied outcomes are actually operationalized comparably.
Context May Be the Explanation Rather Than Noise
A finding that changes across settings may expose a boundary condition.
Perhaps an educational intervention works when teachers receive extensive training but not when implementation support is minimal. Perhaps a workplace policy produces different effects in large and small organizations. Perhaps an intervention's effectiveness depends on baseline resources.
In these situations, averaging all studies into one conclusion can conceal useful structure.
Cochrane cautions that a random-effects meta-analysis does not make heterogeneity disappear. The average effect describes a distribution of effects under model assumptions, but substantial between-study variation still requires interpretation.
Statistical Heterogeneity Can Help, but It Does Not Diagnose the Cause
In quantitative syntheses, statistics such as I2, tau-squared, tests of heterogeneity, and prediction intervals can help characterize between-study variation.
Cochrane warns against interpreting I2 using rigid thresholds because its importance depends on the magnitude and direction of effects and the strength of evidence for heterogeneity. Uncertainty in I2 can also be substantial when few studies are available.
More importantly, a heterogeneity statistic does not tell you why studies differ. That requires substantive investigation.
Explore Explanations Carefully
Potential explanations may include:
| Possible source of disagreement |
What to compare |
What the pattern might reveal |
| Population |
Age, baseline characteristics, risk, occupation, geography |
Effect modification or limited generalizability |
| Intervention or exposure |
Intensity, duration, implementation, definition |
Different versions may produce different effects |
| Outcome |
Construct, instrument, data source, timing |
Studies may not be measuring the same consequence |
| Study design |
Experimental versus observational, longitudinal versus cross-sectional |
Bias or inferential differences may affect estimates |
| Risk of bias |
Confounding, missing data, selective reporting, measurement problems |
Some estimates may be systematically distorted |
| Context |
Setting, resources, implementation environment, period |
The effect may genuinely vary by context |
| Sampling variation |
Effect estimates and uncertainty intervals |
Apparent disagreement may be compatible with chance variation |
Subgroup analyses and meta-regression can sometimes help investigate heterogeneity, but Cochrane emphasizes that such analyses have substantial pitfalls. Explanations developed after seeing the results are particularly vulnerable to spurious patterns and should usually be treated as hypothesis-generating rather than definitive.
Contradiction Can Reduce Certainty in the Evidence
Within GRADE, unexplained inconsistency is one reason certainty in a body of evidence may be reduced. The underlying logic is straightforward: if credible studies produce materially different answers and you cannot explain why, confidence in one simple summary conclusion should decrease.
This connects contradictory evidence directly to where the overall evidence remains weak. Contradiction is one possible reason for weakness, not a synonym for weak evidence itself.
Do Not Vote-Count Positive and Negative Studies
A common approach is to count how many studies report positive, negative, or non-significant findings and declare whichever category has the most studies the winner.
This discards study size, precision, risk of bias, effect magnitude, and the compatibility of estimates. Ten tiny studies do not automatically outweigh one large, rigorous study simply because ten is a larger number than one.
When studies are sufficiently comparable, appropriate quantitative synthesis may be informative. When they are not, a structured synthesis should preserve the reasons they differ rather than forcing them into a crude vote.
Contradiction Can Be Scientifically Productive
Conflicting evidence is not merely a nuisance to be eliminated. It can reveal where a theory's boundary lies, which population responds differently, which implementation condition matters, or which measurement strategy changes the result.
A particularly useful research question often emerges as:
Under what conditions does finding X occur, and what explains why credible studies produce different results?
That is considerably more informative than another generic attempt to determine whether X is “significant.”
