03 · What You Need to Know
How Registered Reports Evaluate Findings Regardless of Their Direction
Why Were Registered Reports Designed to Accommodate Null Results?
Scientific studies do not always support the hypotheses that motivated them. Nevertheless, research publishing has historically faced concerns about publication bias, including the tendency for statistically significant or apparently positive findings to receive greater attention and, in some circumstances, a greater likelihood of publication.
When the published literature disproportionately represents favorable findings, readers may obtain a distorted impression of the available evidence. This can affect systematic reviews, meta-analyses, theoretical development, and subsequent research decisions.
Registered Reports seek to reduce this problem by evaluating the importance of the research question and the adequacy of the proposed methods before the principal results are known.
Following successful Stage 1 peer review, the journal may grant in-principle acceptance. The publication commitment is then conditional on satisfactory implementation and reporting rather than confirmation of the original hypothesis.
This does not eliminate all publication bias or guarantee that every Registered Report will be completed and published. It does, however, change the relationship between the eventual findings and the editorial decision.
What Is the Difference Between Null, Nonsignificant, and Negative Results?
These expressions are sometimes used interchangeably, but they can describe different statistical and substantive situations.
| Expression |
Typical meaning |
Important limitation |
| Statistically nonsignificant result |
A test does not cross its specified significance threshold. |
Does not demonstrate that the null hypothesis is true. |
| Null finding |
A result described as showing no detected effect or association. |
May be inconclusive if uncertainty is substantial. |
| Negative result |
An ambiguous term that may mean nonsignificance or an effect in an unfavorable direction. |
Its intended meaning should be stated explicitly. |
| Opposite-direction finding |
The estimated effect is in the direction opposite to the prediction. |
Does not necessarily establish a genuine reversed effect. |
| Evidence of equivalence |
An appropriate analysis supports effects being within prespecified equivalence bounds. |
Requires a defensible equivalence margin and suitable inferential procedure. |
For example, a treatment effect estimate close to zero with a very wide confidence interval may be compatible with meaningful benefit and meaningful harm. Describing that result as proof that the treatment does nothing would be misleading.
By contrast, a sufficiently precise estimate may rule out effects large enough to matter practically, even if it does not support the original prediction.
Does In-Principle Acceptance Protect a Study With Nonsignificant Findings?
Yes, within the conditions established by the journal.
In the conventional Registered Report model, in-principle acceptance means that the journal has already evaluated the proposed research question and methodology without knowing the main results.
The completed manuscript is then evaluated at Stage 2 for matters such as protocol adherence, implementation integrity, reporting completeness, and appropriate interpretation.
A statistically nonsignificant primary result should not, by itself, justify rejection after IPA.
However, the journal may still identify serious problems, such as unexplained deviations from the approved design, inadequate implementation, or conclusions unsupported by the evidence.
The distinction is central to understanding the conditions attached to in-principle acceptance. IPA protects results-independent evaluation; it does not remove scientific accountability.
How Should Researchers Interpret a Nonsignificant Primary Result?
Begin with the estimated effect and its uncertainty rather than the significance threshold alone.
Suppose a study estimates an achievement difference of 0.8 points between instructional conditions, with a 95% confidence interval from −2.5 to 4.1 points.
The estimate is compatible with several possibilities, including a small disadvantage, no effect, and a potentially meaningful benefit. Whether the result is informative depends on the scale, the practical importance of those differences, and the assumptions underlying the interval.
Researchers should distinguish failure to reject a null hypothesis from evidence supporting a sufficiently small or negligible effect.
When the research question concerns whether an effect is practically equivalent to zero, an equivalence test with justified bounds may be appropriate. Such a test should ideally be planned prospectively rather than selected because an ordinary significance test was nonsignificant.
Can Equivalence Testing Provide Evidence That an Effect Is Negligible?
Yes, when the equivalence margin is substantively justified and the statistical procedure is appropriate.
Equivalence testing evaluates whether the effect can reasonably be considered small enough to fall within a prespecified interval of practically negligible values.
For a two-sided equivalence test using the two one-sided tests procedure at a 5% significance level, equivalence is established when the corresponding 90% confidence interval lies entirely within the prespecified equivalence bounds, under the relevant assumptions.
Equivalence testing should not be used merely to reframe an uninformative nonsignificant result as a successful finding. The margin, estimand, and analysis must be justified by the research question.
Can a Registered Report Publish Findings Opposite to Its Hypothesis?
Yes. A hypothesis-disconfirming result may be scientifically informative, particularly when the study provides a credible test of the proposed explanation.
For example, researchers may predict that frequent automated feedback improves achievement, but the estimated effect may suggest lower achievement among students receiving that feedback.
The finding should be reported accurately, including its uncertainty and any limitations that could explain the pattern.
Researchers should not rewrite the original hypothesis to make the result appear anticipated. Nor should they assume that an opposite-direction estimate necessarily establishes harm.
A credible unexpected result may motivate investigation of mechanisms, contextual factors, or alternative explanations. Those investigations should be distinguished from the prospectively approved tests.
What Happens if a Secondary Outcome Is Significant but the Primary Outcome Is Not?
The primary outcome remains the principal prespecified basis for evaluating the main research objective.
If a secondary outcome produces a statistically significant finding, it may still be scientifically relevant. Its interpretation depends on whether the analysis was prespecified, how multiplicity was addressed, and the strength of the evidence.
Researchers should not retrospectively redefine the secondary outcome as primary simply because its result is favorable.
The distinction is particularly important when the primary outcome has been preregistered and reviewed at Stage 1.
Appropriate reporting may conclude that the study did not provide convincing evidence for the principal hypothesis while identifying potentially informative secondary findings.
Can Unexpected Findings Lead to Additional Analyses?
Yes. Registered Reports can accommodate additional exploratory analyses, subject to the journal's policies.
Unexpected results may suggest mechanisms, subgroup differences, or relationships not anticipated in the approved protocol.
Investigating these possibilities can contribute to scientific understanding. However, analyses developed after examining the data should not be presented as though they were prospectively approved confirmatory tests.
For example, an unexpected association between feedback frequency and achievement might motivate an exploratory analysis of engagement patterns. The analysis could generate a hypothesis for a later study without establishing that hypothesis conclusively.
The relevant distinction is explained through exploratory analyses in Registered Reports.
Do Registered Reports Eliminate Publication Bias?
They are designed to reduce important sources of publication bias, but complete elimination should not be assumed.
Editorial decisions at Stage 1 may still favor particular research questions, methodologies, disciplines, or theoretical perspectives. Studies may also fail to reach completion or Stage 2 publication for reasons unrelated to their results.
Moreover, Registered Reports do not automatically eliminate selective interpretation, measurement problems, or weaknesses in the approved design.
Research comparing Registered Reports with conventional publications has found differences in the frequency of hypothesis-supporting findings, although such comparisons require caution because studies and journals may differ in other ways.
The defensible conclusion is that Registered Reports provide a structural mechanism for reducing results-dependent publication decisions, not that they guarantee an unbiased scientific literature.
What Should the Discussion Section Say When the Hypothesis Is Unsupported?
The discussion should explain what the observed evidence does and does not establish.
Authors should report the primary estimates, uncertainty, relevant assumptions, and limitations without treating statistical nonsignificance as either failure or proof of no effect.
Where appropriate, the discussion may consider why the findings differ from theoretical expectations. Such explanations should be framed as interpretations or hypotheses unless the study directly tested them.
Researchers should also avoid exaggerating unexpected secondary findings to compensate for an unsupported primary hypothesis.
The aim is to provide an accurate account of what the approved investigation contributed to knowledge.
Can a Registered Report With Null Findings Still Be Rejected?
Yes, but null findings alone should not be the reason under the conventional IPA model.
Stage 2 rejection may remain possible when the researchers fail to satisfy important acceptance conditions, when serious implementation problems compromise the study, or when research-integrity concerns arise.
For example, an unexplained change to the primary outcome after examining results may raise a different concern from a properly conducted study whose primary estimate is near zero.
Researchers should preserve the approved protocol and document any changes made after Stage 1 acceptance.
Watch Out
Do not confuse a statistically nonsignificant result with evidence of equivalence or absence of effect. A study can fail to reject the null hypothesis while remaining compatible with effects large enough to matter. Interpretation must consider uncertainty and the design's capacity to provide informative evidence.
