03 · What You Need to Know
The Research Decisions That Matter Most for Preregistration
Why Some Decisions Need More Advance Specification Than Others
Researchers routinely make decisions that can influence empirical findings. They choose which observations to analyze, which outcomes to emphasize, how variables should be constructed, and which statistical models to estimate.
Many of these choices are individually defensible. The concern arises when researchers can select among them after observing which option produces a preferred result. This creates opportunities for selective analysis and reporting, even without deliberate misconduct.
Prespecification limits ambiguity about what was planned before relevant evidence became available. It does not guarantee valid research, eliminate bias, or replace methodological judgment. Rather, it helps readers distinguish prospective tests from decisions influenced by observed findings.
A practical principle is to prioritize choices that could materially alter the estimated effect, statistical uncertainty, study population, or interpretation of a hypothesis.
1. Research Questions and Confirmatory Hypotheses
Begin by specifying the precise research questions and hypotheses that will be evaluated. A hypothesis should identify the expected relationship, comparison, or effect sufficiently clearly to distinguish supporting evidence from competing possibilities.
For example, stating that digital feedback influences learning is less informative than predicting that students receiving individualized digital feedback will obtain higher posttest scores than students receiving standardized feedback.
Where appropriate, specify whether a hypothesis is directional, which groups or variables are being compared, and what result would count as evidence relevant to the prediction.
Distinguish confirmatory hypotheses from exploratory questions. You may plan exploratory investigations in advance without presenting them as fully specified hypothesis tests.
2. Study Design, Comparison Groups, and Assignment Procedures
Prespecify the features of the design that determine how evidence will be generated and what inferences it can support.
For experimental studies, these commonly include the intervention conditions, comparison group, unit of randomization, allocation procedure, and any planned blocking or stratification. Blinding procedures should also be described when applicable.
For observational research, identify the population, exposure or predictor variables, comparison structure, measurement timing, and intended approach to addressing confounding.
For longitudinal studies, specify measurement occasions and the principal temporal relationships of interest. For clustered designs, identify the levels at which participants are recruited, assigned, measured, and analyzed.
These details matter because a statistical analysis cannot compensate automatically for weaknesses in the underlying design. Preregistration also does not convert an observational association into evidence of causation.
3. Primary Outcomes and Their Operational Definitions
The primary outcome deserves particularly careful specification because it establishes the principal basis for evaluating the study's central question.
Identify what will be measured, how it will be measured, when it will be measured, and how the recorded information will be converted into an analyzable variable.
For example, a study of learning achievement might specify a 30-item assessment administered immediately after an intervention, scored as the total number of correct answers from 0 to 30.
If a scale contains several items, describe the scoring rule, reverse coding, aggregation method, and handling of incomplete responses. If several outcomes are designated as primary, specify how their results will be evaluated together.
Secondary outcomes should be identified separately. Without this distinction, researchers may have considerable flexibility to emphasize whichever measure produces the most favorable finding.
4. Target Population, Inclusion Criteria, and Exclusion Criteria
Specify the intended study population and the conditions determining which observations will enter the analysis.
This includes participant inclusion and exclusion criteria, such as eligibility characteristics, prior exposure, minimum participation requirements, or relevant recruitment restrictions.
Distinguish eligibility rules from exclusions applied after data collection. A participant may satisfy recruitment eligibility but later be excluded from a particular analysis because of an explicitly defined data-quality issue.
Rules should be operational rather than subjective. For example, "exclude low-quality responses" leaves considerable discretion. A specific criterion defining invalid responses and its intended application is more reproducible.
Criteria should be justified by the research question and study design. Exclusions that are unnecessary or excessively restrictive may reduce generalizability or introduce selection bias.
5. Sample Size, Sampling Strategy, and Stopping Rules
Prespecify the intended number of participants or observational units, the rationale for that number, and the conditions under which recruitment or data collection will stop.
The relevant unit may be a participant, classroom, school, hospital, organization, or another entity. In multilevel studies, the number of units at each important level may need to be specified.
Sample-size justification may involve statistical power, precision, expected event rates, feasibility, or another design-appropriate rationale. Not every study requires the same calculation.
A clear sample size and stopping rule also explains whether the target refers to recruited participants, completed observations, or the final analyzable sample.
For studies using interim analyses or adaptive sampling, specify the decision rules and appropriate statistical framework. Repeatedly checking results and stopping when statistical significance appears is not equivalent to a prospectively justified sequential design.
6. Data Cleaning, Exclusions, and Missing-Data Procedures
Data preparation can materially influence results. Researchers should therefore prespecify consequential rules for identifying invalid observations, managing missing values, and transforming recorded data.
Examples include handling duplicate records, defining impossible measurements, identifying technical failures, and specifying whether particular observations will be retained, corrected, or excluded.
For data exclusion rules, distinguish participant-level exclusions from exclusions of individual measurements.
Missing-data plans should describe the intended method, such as complete-case analysis, multiple imputation, or an appropriate likelihood-based approach, when relevant. The choice should reflect plausible missingness mechanisms and the requirements of the statistical model.
Where assumptions cannot be assessed fully before data collection, researchers may specify conditional procedures. For example, the plan can state what will happen if a predefined level of missingness occurs or if a particular data-quality problem is detected.
7. Statistical Models and the Exact Confirmatory Tests
Simply writing "the data will be analyzed using regression" is usually insufficient for a confirmatory analysis plan.
Identify the outcome variable, predictors, planned covariates, model family, relevant interactions, and the statistical parameter or contrast used to evaluate each hypothesis.
Depending on the design, additional specifications may include:
- The link function and distributional assumptions for generalized models.
- Random effects or correlation structures for repeated or clustered observations.
- Variable transformations and centering procedures.
- Planned interaction terms and subgroup comparisons.
- Methods for estimating uncertainty, such as confidence intervals or appropriate standard errors.
- Criteria for statistical inference and any adjustment for multiple testing.
Not all of these are relevant to every study. The objective is to specify the choices that determine how the principal analysis will actually be conducted.
8. Covariates, Confounders, and Adjusted Analyses
Covariate selection can substantially change estimated relationships. Identify the variables that will enter the primary adjusted model and explain their intended role.
For randomized experiments, baseline covariates may improve precision when chosen appropriately. For observational studies, adjustment decisions are often closely tied to assumptions about confounding and causal structure.
A variable should not be included merely because it happens to correlate with the outcome in the collected data. Conversely, excluding a theoretically important confounder because it is statistically nonsignificant may undermine the intended analysis.
If several models will be reported, specify which is primary and which serve as secondary or sensitivity analyses.
9. Subgroup Analyses, Interactions, and Multiple Comparisons
When researchers expect an effect to differ across participant groups, planned subgroup analyses should identify the grouping variable, the comparison, and the statistical procedure used to assess heterogeneity.
Finding statistical significance in one subgroup but not another does not, by itself, establish that the groups differ. A formal interaction or other appropriate comparison is generally needed.
Also identify families of confirmatory tests and the approach to multiplicity where relevant. Testing numerous outcomes, subgroups, and alternative models without accounting for the resulting inferential complexity can increase the risk of misleading conclusions.
Exploratory subgroup analyses remain legitimate, but their evidential status should be communicated accurately.
10. Assumption Checks, Contingencies, and Sensitivity Analyses
Researchers cannot anticipate every property of future data. Nevertheless, they can often specify how foreseeable problems will be handled.
For example, a plan may define the diagnostic evidence that would prompt use of an alternative estimator or specify how a robustness analysis will evaluate sensitivity to particular modeling assumptions.
Such contingencies should be explicit enough to constrain discretion without pretending that a single diagnostic threshold can resolve every methodological problem.
Sensitivity analyses should also be distinguished from unrestricted model searching. Prespecified alternatives can help readers understand how robust the conclusions are to defensible assumptions.
How Detailed Should a Prespecified Plan Be?
A useful test is whether two competent analysts, working independently from the same data and preregistration, would be likely to implement substantially the same primary analysis.
They need not produce identical code, but they should not have to invent important choices concerning outcomes, exclusions, covariates, model specifications, or inferential thresholds.
The appropriate degree of specificity varies. A tightly controlled experiment may permit detailed advance specification. A qualitative inquiry, an emerging-methods study, or an exploratory investigation may reasonably preserve greater flexibility.
For qualitative research preregistration, the emphasis may fall on the research orientation, sampling rationale, data generation, analytic approach, and intended documentation of evolving interpretations rather than fixed statistical tests.
Watch Out
More detail does not automatically mean better preregistration. A highly specific plan can still encode a biased design, inappropriate model, or unjustified exclusion rule. Prespecification improves transparency about decisions; it does not establish that those decisions are scientifically sound.
