03 · What You Need to Know
Why Statistical Nonsignificance and Equivalence Answer Different Questions
What Does “No Statistically Significant Difference” Actually Mean?
In a conventional two-sided significance test, the null hypothesis generally states that the population difference is zero. The alternative hypothesis states that the difference is not zero.
When the p-value exceeds the prespecified significance level, researchers fail to reject the null hypothesis.
This means the observed data do not provide sufficient evidence, under the statistical model and testing procedure, to reject the specified null hypothesis at that threshold.
It does not establish that the null hypothesis is true. Nor does it determine whether the range of effects compatible with the data is substantively acceptable.
A p-value of 0.40, for example, does not mean there is a 40% probability that the treatments are equivalent. A conventional p-value does not directly provide the probability that a hypothesis is true.
Why Can an Underpowered Study Produce a Nonsignificant Result?
Statistical power describes the probability that a test will reject its null hypothesis under a specified alternative and set of assumptions.
When a study has insufficient information, its effect estimate may be imprecise. The resulting confidence interval can include zero while also extending across differences that would be important in practice.
For example, a small trial might estimate no treatment difference but produce a confidence interval ranging from a substantial disadvantage to a substantial benefit.
The study has not established similarity. It has failed to distinguish among several possibilities.
Increasing sample size can improve precision under appropriate assumptions, but sample size alone does not guarantee validity. Bias, inappropriate measurement, and confounding can remain even in very large studies.
What Does Equivalence Testing Ask Instead?
Equivalence testing asks whether the population effect lies within a prespecified region of acceptable differences.
Rather than testing only whether the effect differs from zero, researchers define lower and upper boundaries representing differences considered too large for the intended application.
Suppose the effect is the mean difference between two measurement procedures. Researchers might specify that differences between −3 and +3 units are acceptable.
To demonstrate equivalence, the evidence must support the conclusion that the population difference lies strictly within those boundaries.
This is the central distinction between ordinary nonsignificance and a properly designed equivalence study.
How Do the Hypotheses Differ?
Let θ represent the population difference between two interventions or methods. Let L and U represent the lower and upper equivalence boundaries, with L smaller than U.
Conventional two-sided difference test
H₀: θ = 0; H₁: θ ≠ 0. The test seeks evidence against an exact zero difference.
Equivalence test
H₀: θ ≤ L or θ ≥ U; H₁: L < θ < U. The test seeks evidence that the difference lies within acceptable boundaries.
These hypotheses place the burden of evidence in different locations.
In conventional difference testing, researchers need evidence to reject zero. In equivalence testing, they need evidence to reject differences at or beyond the unacceptable boundaries.
Consequently, the same dataset may produce a nonsignificant conventional test while also failing to demonstrate equivalence.
How Do Confidence Intervals Reveal the Difference?
A confidence interval provides an estimate of the uncertainty surrounding an effect under the specified statistical procedure.
For a conventional two-sided superiority test at α = 0.05, a corresponding 95% confidence interval that includes zero generally indicates that the difference is not statistically significant.
However, an interval containing zero may be narrow or extremely wide.
If the interval extends beyond the equivalence margins, the study has not excluded differences that were defined as unacceptable.
For a conventional two one-sided tests procedure at α = 0.05 for each component test, equivalence is established when the corresponding two-sided 90% confidence interval lies entirely within the prespecified margins.
The confidence level must match the testing procedure. Researchers should not automatically substitute a 95% interval for a 90% interval without recognizing that this changes the criterion.
What Is the Two One-Sided Tests Procedure?
The two one-sided tests procedure, commonly abbreviated TOST, evaluates two separate requirements.
The first test evaluates whether the population effect is sufficiently above the lower equivalence boundary. The second evaluates whether it is sufficiently below the upper boundary.
Both requirements must be satisfied for equivalence to be demonstrated.
The calculation does not establish that individual observations or measurements are interchangeable. It addresses the specified population effect, such as a mean difference.
Can a Study Find Both a Statistically Significant Difference and Equivalence?
Yes. This is possible because statistical significance and equivalence evaluate different properties of an effect.
Imagine a very precisely estimated mean difference of +0.5 units. A conventional test may establish that the difference is not zero.
If the corresponding equivalence confidence interval lies entirely within justified margins of −3 and +3 units, the study may also demonstrate equivalence.
There is no contradiction. The effect can be detectably different from zero while remaining too small to matter for the intended application.
Statistical significance does not automatically imply substantive importance, just as statistical nonsignificance does not establish substantive similarity.
Can a Nonsignificant Result Ever Be Consistent With Equivalence?
Yes. A nonsignificant conventional test and a successful equivalence test can occur in the same study.
For example, a narrow confidence interval centered near zero may provide insufficient evidence to reject an exact zero difference while still excluding all differences beyond the equivalence boundaries.
The crucial point is that the nonsignificant p-value does not establish equivalence. The separate equivalence analysis does.
If the margins were not prespecified, a post hoc assessment may be informative but should not be represented as a planned confirmatory demonstration.
How Is Equivalence Different From Noninferiority?
Noninferiority is concerned with excluding an unacceptable disadvantage in one direction. Equivalence requires excluding unacceptable differences in both directions.
A treatment may satisfy a noninferiority margin while producing a substantial improvement over the comparator. That improvement ordinarily does not undermine the noninferiority objective.
For equivalence, however, an effect beyond the upper boundary may mean that the treatments are not equivalent according to the prespecified definition.
The distinction between superiority, noninferiority, and equivalence should be established before selecting the hypothesis test.
Why Must Equivalence Margins Be Substantively Justified?
Equivalence testing can demonstrate acceptable similarity only relative to its chosen boundaries.
If researchers choose extremely wide margins, almost any reasonably precise comparison might satisfy them. The resulting conclusion could be statistically correct but practically uninformative.
Margins should therefore reflect the largest differences that would be acceptable for the intended use.
In clinical research, these judgments may depend on patient outcomes and treatment consequences. In educational research, they might depend on a defensible threshold for meaningful differences in learning outcomes or implementation performance.
In measurement research, the acceptable limits should reflect the decisions that depend on the measurements.
Importantly, equivalence of average scores or mean measurements does not necessarily imply equivalence for every individual participant or observation.
What Should Researchers Report When the Evidence Is Inconclusive?
When a conventional test is nonsignificant and equivalence has not been demonstrated, the defensible conclusion is that the study did not establish a difference under the conventional test and did not provide sufficient evidence of equivalence.
This may occur because the confidence interval is too wide, the equivalence margins are restrictive, or the data are affected by methodological limitations.
Researchers should report the effect estimate, confidence interval, relevant margins, and the limitations affecting interpretation.
They should avoid replacing an inconclusive finding with an unsupported declaration that the interventions are equally effective.
06 · What This Means for You
How to Interpret and Report Results Without Overclaiming Equivalence
When a conventional hypothesis test produces a nonsignificant result, examine the effect estimate and confidence interval before drawing conclusions.
Ask whether the interval excludes differences that would matter for the research question. If meaningful effects remain compatible with the data, the study cannot establish that those effects are absent.
A simple decision framework
If the original objective was to detect a difference
Report the estimated effect, uncertainty, and whether the prespecified difference test rejected its null hypothesis.
If the objective was to establish acceptable similarity
Use prespecified equivalence margins and an appropriate equivalence test.
If the confidence interval crosses an equivalence boundary
Conclude that equivalence was not established, rather than asserting that the interventions are different or identical.
If equivalence was not planned before examining results
Treat any subsequent equivalence assessment as exploratory and disclose how its margins were selected.
For future studies, determine whether the research question concerns a difference, an unacceptable disadvantage, or acceptable similarity before calculating sample size.
When only a one-directional disadvantage must be excluded, noninferiority may be more appropriate than equivalence.
When the intended decision requires acceptable similarity in both directions, justify the equivalence margins and design the study to estimate the relevant effect with sufficient precision.
Watch Out
Do not use the expressions no effect, equally effective, equivalent, or interchangeable as substitutes for no statistically significant difference. These statements make different claims and require different evidence.
