How Do You Recognize When Authors Shift From “No Evidence” to “No Effect”?

Does failing to detect an effect mean there is no effect?

A study compares two groups and obtains p =.12. The Results say that the difference was not statistically significant. The Discussion says there was “no evidence of a difference.” Then the Conclusion says the intervention “has no effect.”

Those statements are not necessarily equivalent.

A study can fail to provide convincing evidence for an effect because the true effect is small or absent. But the same outcome can occur when the data are too imprecise to distinguish no effect from an effect large enough to matter. Critical reading requires you to determine which situation the evidence actually supports.

Failure to establish an effect is not automatically evidence of absence

Why “not statistically significant” does not mean “no effect”

Start with three different statements

The second and third situations are particularly easy to confuse. A study can be inconclusive precisely because it cannot tell you whether an important effect exists.

Read the effect estimate before the significance label

Suppose a trial estimates that an intervention improves a score by 3 points, with a 95% confidence interval from -2 to 8 points.

If the conventional null value for the difference is zero, this interval includes zero. A conventional superiority test may therefore be statistically non-significant.

But the interval also includes a possible improvement of 8 points. If an 8-point improvement would matter substantially, the data have not ruled out an important benefit.

Calling the result “no effect” would erase that uncertainty.

CONSORT 2025 specifically warns against interpreting a non-significant result as equivalence and emphasizes confidence intervals because they can show that a statistically non-significant result remains compatible with an important effect.

A p-value does not estimate how large the effect is

The American Statistical Association cautions that a p-value or statistical-significance designation does not measure effect size or the importance of a result. Scientific conclusions should not be based only on whether a p-value passes a threshold.

This matters because p-values depend partly on the amount and variability of information in the study. Similar estimated effects can produce different p-values in studies with different precision.

Therefore:

p >.05 does not translate mathematically into “effect = 0.”

Nor does p <.05 mean the effect is large or important. This is one reason to keep statistical significance separate from practical importance.

Wide uncertainty makes “no effect” especially problematic

Imagine two studies with the same point estimate of zero.

Both point estimates are zero. Both studies might produce a non-significant test against a null difference. Yet they provide very different information.

Study B tells you far less about whether an important effect is absent. This is why reading only “not significant” discards information you need.

Look for the linguistic shift across the paper

The error often appears gradually:

Results: “The difference was not statistically significant.”

Discussion: “We found no evidence that the intervention improved performance.”

Conclusion: “The intervention does not improve performance.”

The first statement reports the result of a statistical decision rule. The second may be a reasonable summary if appropriately understood. The third makes a stronger claim about absence.

Place these sentences next to each other. If certainty increases while the underlying evidence remains unchanged, ask what justifies the stronger conclusion.

“No evidence of X” can itself be misunderstood

Even the phrase “no evidence of an effect” needs care. It can sound as though the study produced affirmative evidence that no effect exists.

Sometimes a more informative statement is simply to report the estimate and uncertainty:

“The estimated difference was 1.4 points, with a 95% confidence interval from -2.1 to 4.9 points.”

This lets readers see what the data remain compatible with instead of compressing the entire result into a binary label.

Evidence of absence requires a meaningful definition of “absence”

In most applied research, the scientifically interesting question is rarely whether the true effect is exactly zero to infinite precision.

A more useful question may be whether any remaining effect is small enough to be considered practically unimportant.

That requires a margin or criterion. How small would a difference need to be before you would treat the interventions as practically equivalent? What difference would matter educationally, clinically, economically, or otherwise?

Without such a criterion, “no meaningful effect” can become an undefined claim.

Equivalence testing asks a different question from ordinary superiority testing

A conventional superiority analysis typically asks whether the data provide sufficient evidence of a difference under a specified null hypothesis. Failing to demonstrate superiority does not reverse the test and demonstrate equivalence.

An equivalence design instead asks whether the effect is sufficiently bounded within a prespecified equivalence region. The equivalence margin should have a substantive justification rather than being selected after the results are known merely to obtain the desired conclusion.

Similarly, a non-inferiority trial asks whether an intervention can be shown not to be worse than a comparator by more than a prespecified margin. CONSORT 2025 treats superiority, equivalence, and non-inferiority as distinct trial frameworks that should be explicitly identified.

Therefore, if a conventional superiority trial obtains p >.05 and the authors declare the treatments “equivalent,” check whether an actual equivalence analysis was planned and conducted.

Sample size matters through precision, but “underpowered” is not a complete interpretation

A small sample can produce imprecise estimates, making it difficult to distinguish meaningful effects from no effect. But simply labeling a study “underpowered” after seeing a non-significant result is not enough.

Inspect the effect estimate and its uncertainty directly. The confidence interval can show which effect sizes remain compatible with the data.

A study with a smaller-than-planned sample might still provide useful information if its estimate is sufficiently precise for the question at hand. Conversely, a nominally large study can leave important uncertainty for rare outcomes or noisy measurements.

Precision is the evidential issue you ultimately care about.

Evidence of absence is possible

The slogan “absence of evidence is not evidence of absence” is a warning, not a rule that absence can never be supported.

Studies can provide evidence against effects of a specified meaningful magnitude. Equivalence tests, well-designed non-inferiority analyses, sufficiently precise confidence intervals, and some Bayesian approaches can all contribute to such conclusions when used appropriately.

The critical distinction is between failing to detect something and obtaining evidence that meaningfully constrains how large it could be.

Compare “no significant difference” with evidence of practical equivalence

This is why the significance label alone cannot tell you whether you have no evidence of an effect or useful evidence that a meaningful effect is absent.

Common mistakes when interpreting null and inconclusive findings

Replace binary significance reading with an uncertainty check

Whenever a paper claims that something “does not work,” “has no effect,” “makes no difference,” or is “equivalent,” do not begin with the p-value. Begin with the inferential question and the range of effects still compatible with the evidence.

This is another reason to keep the numerical result separate from the authors’ narrative. A binary sentence can conceal a great deal of uncertainty.

It also prevents an inconclusive finding from becoming one of the claims introduced more strongly in the Conclusion than the Results support.

Did “not detected” quietly become “does not exist”?

Questions about absence of evidence and evidence of absence

Inconclusive evidence and evidence of absence are different results

How to Cite This Guide

This guide is intended to be read, shared, and used in research, teaching, and academic work. If you draw on its ideas, explanations, or other content, please acknowledge the source by citing the guide. Doing so gives appropriate credit and helps your readers locate the original resource.