What Do Reviewers Evaluate During Stage 1 Peer Review of a Registered Report?

What Are Reviewers Looking for Before Your Study Produces Results?

You have submitted a Stage 1 Registered Report containing your introduction, research questions, hypotheses, methods, and proposed analyses. Unlike conventional manuscript review, however, the reviewers cannot evaluate whether your principal findings support the hypotheses because those results are not yet available.

What, then, determines whether the manuscript is accepted? Are reviewers primarily assessing the novelty of the question, the statistical power of the design, the completeness of the protocol, or the likelihood that the expected findings will be significant?

Understanding Stage 1 evaluation matters because reviewers are assessing the scientific value and credibility of a proposed investigation. Their task is to determine whether the research question warrants investigation and whether the planned methods can answer it convincingly, independently of the eventual direction of the findings.

Reviewers Evaluate the Question, Design, and Evidential Value of the Plan

The Main Criteria Used in Stage 1 Registered Report Review

How Does Stage 1 Review Differ From Conventional Peer Review?

In conventional peer review, a completed manuscript ordinarily contains the research rationale, methods, results, and interpretation. Reviewers can evaluate the reported findings alongside the methodological decisions that produced them.

Stage 1 review occurs earlier. The manuscript normally presents the scientific rationale and proposed methods before the main results are known. Reviewers therefore concentrate on whether the investigation is worth conducting and whether the proposed evidence will be capable of addressing the research question.

The distinction changes the timing of methodological criticism. Instead of identifying a major design weakness after data collection, reviewers may recommend a correction while the design can still be improved.

Stage 1 review is not inherently more rigorous than every conventional review. Its distinctive advantage is that methodological scrutiny occurs before many consequential decisions become difficult or impossible to change.

1. Is the Research Question Important and Sufficiently Justified?

Reviewers first need to understand why the proposed question deserves investigation.

The introduction should establish the relevant theoretical or empirical problem, identify what remains uncertain, and explain how the proposed study could advance understanding.

Importance does not necessarily require dramatic novelty. A carefully designed replication, a test of competing explanations, or a study addressing a consequential uncertainty may have substantial scientific value even without introducing a new theory.

Reviewers may question a proposal when the rationale is weak, the question is already adequately resolved, or the proposed evidence would add little meaningful information.

Authors should therefore explain the contribution of the study without implying that its value depends on obtaining a particular result.

2. Can the Proposed Design Actually Answer the Research Question?

A central Stage 1 concern is the alignment between the research question and the proposed design.

For an experimental study, reviewers may examine the intervention, comparison conditions, allocation procedures, unit of randomization, blinding arrangements, and potential sources of contamination.

For observational research, attention may focus on the sampling frame, temporal ordering, measurement, confounding, selection processes, and the assumptions required for the intended interpretation.

For longitudinal or multilevel research, reviewers may assess whether the timing and structure of observations support the proposed analysis.

Importantly, a sophisticated statistical model cannot automatically correct a design that does not identify the intended effect or relationship.

Authors should be able to explain what evidence the study would produce and which conclusions that evidence could reasonably support.

3. Are the Hypotheses and Research Objectives Sufficiently Clear?

When a Registered Report is confirmatory, reviewers generally expect hypotheses to be stated precisely enough that the planned tests can be identified before results are examined.

A statement such as "technology will influence learning" leaves substantial ambiguity. A more informative hypothesis identifies the intended comparison, outcome, and predicted relationship.

However, not every Registered Report requires directional hypotheses. Some qualitative, descriptive, or other methodologically appropriate submissions may be organized around research questions and specified analytic objectives.

Reviewers should evaluate clarity according to the research approach rather than assume that all credible studies must use the same hypothesis-testing structure.

4. Are the Outcomes and Measurements Appropriate?

Reviewers may examine whether the proposed measurements represent the constructs the researchers intend to study.

For example, if a study claims to evaluate conceptual understanding but uses only a brief satisfaction questionnaire, the outcome may not adequately address the stated objective.

Relevant considerations include measurement validity, reliability where appropriate, scoring procedures, assessment timing, sensitivity to meaningful differences, and potential measurement bias.

For confirmatory studies, reviewers may also assess whether the primary outcome is sufficiently specified and distinguished from secondary or exploratory outcomes.

A prespecified instrument is not necessarily a valid instrument. The manuscript should justify why the proposed measurement is appropriate for the population and research question.

5. Is the Sampling Plan Adequate?

Stage 1 reviewers need to determine whether the planned sample can provide sufficiently informative evidence.

In quantitative hypothesis-testing studies, this may involve examining power calculations, assumed effect sizes, expected variability, allocation ratios, and the statistical consequences of clustering or repeated measurements.

For estimation-focused studies, precision and uncertainty may be more relevant than conventional statistical power.

For qualitative studies, reviewers may evaluate the sampling rationale, intended variation among cases, information needs, and coherence with the methodological approach.

The sample-size justification and stopping rules should be transparent. A numerical target without a defensible rationale may not provide enough information for reviewers to evaluate design sensitivity.

Some journals establish specific statistical or methodological requirements for Registered Reports. Authors should follow those requirements rather than assume that a particular power threshold applies universally.

6. Are the Planned Statistical Analyses Sufficiently Specified?

For confirmatory quantitative studies, reviewers generally expect a sufficiently detailed analysis plan to establish how each principal hypothesis will be evaluated.

The plan should identify relevant outcome variables, predictors, comparisons, covariates, model specifications, and inferential procedures.

Where applicable, it should also explain missing-data treatment, exclusion rules, multiple-testing adjustments, planned interactions, and sensitivity analyses.

For example, writing "the data will be analyzed using regression" may be inadequate if several regression models could reasonably be selected after observing the results.

Reviewers may ask authors to clarify these choices, especially when alternative specifications could materially change the conclusions.

The purpose of prespecifying consequential analysis decisions is to make the intended confirmatory evaluation identifiable before the evidence is examined.

7. Can the Study Produce Informative Results Even if the Hypothesis Is Not Supported?

This is one of the most consequential questions in Stage 1 review.

A study should be capable of producing evidence that meaningfully addresses its research question, including when the findings do not support the predicted effect.

Reviewers may therefore assess whether the design has sufficient sensitivity to distinguish meaningful effects from alternatives, whether the measurements are appropriate, and whether the planned analysis can support a useful interpretation.

A nonsignificant result from a severely underpowered study may remain highly inconclusive. By contrast, a precise estimate inconsistent with effects of practical importance may provide valuable evidence, even without conventional statistical significance.

Informative results do not necessarily require a particular statistical framework. Equivalence testing, interval estimation, Bayesian analysis, or other methods may be appropriate when aligned with the research question.

The central issue is whether the proposed design can yield interpretable evidence rather than merely a favorable p-value.

8. Are Data Exclusions and Methodological Contingencies Addressed?

Reviewers may examine whether the protocol anticipates foreseeable problems in data collection and analysis.

These can include missing outcomes, technical failures, participant nonadherence, invalid measurements, recruitment difficulties, and violations of statistical assumptions.

Consequential data exclusion rules should be sufficiently defined to prevent ambiguity about which observations will enter the principal analyses.

At the same time, a protocol cannot anticipate every problem. Reviewers may value conditional decision procedures that allow scientifically justified responses to foreseeable difficulties.

The aim is not to eliminate methodological judgment but to distinguish planned contingencies from subsequent departures.

9. Is the Proposed Study Feasible and Ethically Appropriate?

A methodologically attractive design is of limited practical value if the researchers cannot implement it.

Reviewers may consider whether recruitment targets are realistic, instruments are available, interventions can be delivered consistently, and the proposed timeline is plausible.

Pilot work, feasibility assessments, or documented access to study sites may strengthen the proposal when relevant.

Ethical considerations can also affect feasibility and design quality. Reviewers may identify concerns involving consent, participant burden, confidentiality, safety, or the management of sensitive information.

However, journal peer review does not replace approval by an institutional ethics committee or another responsible governance authority.

Authors should distinguish editorial evaluation from the formal approvals required before implementing particular research procedures.

10. Are the Planned Interpretations Appropriate to the Evidence?

Some journals ask authors to explain how different possible results would be interpreted.

This may involve identifying what would count as evidence for or against a hypothesis, how uncertainty will be evaluated, and which conclusions the design cannot support.

For example, an observational study may be able to estimate an association but not establish causation without additional assumptions.

Similarly, a statistically nonsignificant result should not automatically be interpreted as proof of no effect.

Reviewers may therefore request clearer interpretation criteria or more appropriate analyses before the study proceeds.

How Do Review Criteria Differ for Qualitative and Mixed-Methods Studies?

Registered Reports are not limited in principle to conventional quantitative experiments, although journal support varies.

For qualitative research, reviewers may assess the coherence of the methodological orientation, sampling rationale, data generation strategy, analytic approach, reflexivity, and ethical safeguards.

They should not automatically require fixed coding categories, directional hypotheses, or statistical power calculations when those requirements conflict with the chosen methodology.

Mixed-methods submissions may additionally require a clear rationale for integrating qualitative and quantitative components, including how findings from one phase will inform subsequent procedures.

The appropriate standards depend on the specific methodology and the journal's capacity to evaluate it. The issues surrounding qualitative and mixed-methods Registered Reports therefore deserve particular attention during journal selection.

What Decisions Can Follow Stage 1 Review?

The precise editorial categories differ among journals, but Stage 1 review may lead to rejection, requests for revision, or in-principle acceptance.

Major revisions are not necessarily evidence that the proposal is fundamentally unsuitable. They may reflect reviewers' attempts to strengthen the study before implementation.

However, reviewers may recommend rejection when the question lacks sufficient justification, the design cannot adequately answer it, or the proposed study is not feasible.

In-principle acceptance establishes a conditional commitment under the journal's procedures. It does not mean that the final manuscript is exempt from Stage 2 evaluation.

The distinction is important when considering the publication conditions attached to in-principle acceptance.

How Should Authors Respond to Stage 1 Reviewer Comments?

Responses should focus on improving the scientific design rather than defending every original decision.

When reviewers identify a valid methodological concern, authors should explain the revision and show how it improves the proposed study.

If a recommendation is inappropriate for the methodology, a respectful, evidence-based explanation may be warranted. Reviewers can disagree about defensible methods, and not every suggestion necessarily improves the research.

A useful response identifies the concern, the action taken, the revised manuscript location, and any remaining methodological justification.

Authors should also ensure that revisions remain consistent across the research questions, methods, analysis plan, and proposed interpretation.

How Stage 1 Review Can Improve an Educational Intervention

The principal benefit in this example is not that reviewers can predict whether AI-supported feedback will improve achievement. It is that they can identify consequential weaknesses before the study is conducted.

The example also illustrates why a large participant count does not necessarily guarantee adequate statistical power when assignment occurs at the cluster level.

Common Misunderstandings About Stage 1 Peer Review

How to Prepare a Stage 1 Manuscript That Can Withstand Methodological Review

Prepare the manuscript as a research proposal whose scientific value must be demonstrated before results are available.

Rather than emphasizing expected significance, explain why the question matters, how the design addresses it, and what the possible findings would contribute to knowledge.

Before submission, consider an independent methodological review by a colleague who has not participated in developing the protocol. Ask whether the proposed study could answer its central question even if the main hypothesis is unsupported.

That question is often more useful than asking whether the proposed study is likely to produce a publishable positive result. The broader benefits of early peer review for research design arise precisely because consequential weaknesses can be addressed before implementation.

Before Submitting a Stage 1 Registered Report

Frequently Asked Questions About Stage 1 Registered Report Review

Stage 1 Review Asks Whether the Proposed Study Deserves to Be Conducted

How to Cite This Guide

This guide is intended to be read, shared, and used in research, teaching, and academic work. If you draw on its ideas, explanations, or other content, please acknowledge the source by citing the guide. Doing so gives appropriate credit and helps your readers locate the original resource.