03 · What You Need to Know
Determining the Appropriate Level of Statistical Analysis Planning
What Is a Statistical Analysis Plan?
A statistical analysis plan, commonly abbreviated SAP, is a document specifying how quantitative research data will be prepared, summarized, analyzed, and interpreted in relation to the study's objectives.
It translates the research protocol into operational analytical decisions.
For example, a protocol may state that an intervention's effect on academic performance will be evaluated. The SAP should clarify which performance outcome is analyzed, which participants are included, what comparison is estimated, how the statistical model is specified, and how uncertainty is quantified.
An SAP is not merely a list of statistical tests. It should explain the analytical logic connecting the research question, design, variables, and intended inference.
The term is particularly established in clinical trials, although equivalent analytical planning is valuable in observational and other quantitative research.
How Does an SAP Differ From a Research Protocol?
Research protocol
Describes the study's objectives, design, participants, procedures, outcomes, ethics, and overall methodological approach.
Statistical analysis plan
Provides operational detail about the analytical populations, variables, estimands, statistical methods, assumptions, uncertainty, and handling of analytical complications.
The documents should be consistent. The SAP may expand the protocol's statistical methods without changing its scientific objectives.
For a relatively simple study, an adequately detailed analytical section within the protocol may be sufficient. A separate SAP is particularly useful when analyses are complex, multiple analysts are involved, or a funder, regulator, sponsor, or institution requires one.
Researchers should verify applicable requirements rather than assume that every study must use an identical document format.
Why Is Statistical Detail Important Before Data Collection?
Many datasets support several plausible analytical choices.
Researchers may choose different outcome transformations, covariates, exclusion rules, subgroup definitions, or missing-data procedures.
When these choices are made after examining their effects on results, the analysis may become vulnerable to selective reporting and exaggerated certainty.
Prespecification helps distinguish decisions justified by the research question and design from decisions influenced by the observed data.
It can also reveal design problems before data collection. For example, planning a multilevel analysis may show that the proposed number of research sites is insufficient for the intended inference.
However, prespecification does not automatically guarantee valid analysis. A precisely documented but inappropriate model remains inappropriate.
How Detailed Should the SAP Be for Different Research Designs?
The required detail should be proportionate to the study's inferential claims and analytical complexity.
| Study type |
Typical planning emphasis |
Important qualification |
| Confirmatory randomized trial |
Primary estimand, analysis population, model, covariates, missing data, sensitivity analyses, and multiplicity |
Applicable trial, sponsor, and regulatory requirements may prescribe additional detail. |
| Observational causal study |
Target population, exposure, outcome, confounding strategy, identification assumptions, and sensitivity analyses |
Prespecification does not eliminate unmeasured confounding. |
| Descriptive survey |
Target quantities, weighting, variable definitions, missingness, uncertainty, and subgroup summaries |
Sampling design determines appropriate population inference. |
| Longitudinal study |
Time structure, repeated-measures model, within-person dependence, attrition, and missing data |
Measurement timing and temporal assumptions must be explicit. |
| Exploratory quantitative study |
Analytical objectives, data processing principles, and transparent reporting of exploratory decisions |
Not every analysis must be fixed before data exploration. |
The same statistical test may require different specifications depending on whether it supports a confirmatory hypothesis, an exploratory association, or a descriptive estimate.
Should the SAP Identify the Exact Research Questions and Hypotheses?
Yes, particularly when the study intends to test specific hypotheses.
Each primary research question should be linked to an appropriate analytical target and method.
For example, asking whether an intervention improves writing performance requires defining the outcome, the comparison conditions, the relevant time point, and the effect being estimated.
A statement that the data will be analyzed using regression is insufficient because it does not identify the intended coefficient, adjustment variables, or interpretation.
Secondary and exploratory objectives should also be distinguished so readers can identify the role of each analysis.
This alignment helps prevent statistical methods from becoming detached from the substantive research purpose.
How Precisely Should Outcomes Be Defined?
Outcomes should be operationally defined to the level necessary for consistent analysis.
The SAP may specify the measurement instrument, scoring procedure, scale, time point, direction of interpretation, and treatment of composite scores.
For example, academic writing performance might be defined as a total score on a prespecified analytic rubric, assessed at the end of an intervention by appropriately trained raters.
If multiple writing assessments are collected, the plan should identify which one represents the primary outcome.
Researchers should also explain whether the analysis concerns final scores, changes from baseline, or another target quantity.
These alternatives may have different statistical properties and should not be selected solely according to which produces the most favorable result.
Should the SAP Define an Estimand?
When a study seeks to estimate an intervention effect or another clearly specified target quantity, the SAP should identify what is being estimated.
In clinical trials, the ICH E9(R1) framework emphasizes defining an estimand through attributes including the treatment condition, population, variable, handling of intercurrent events, and population-level summary.
For example, researchers may seek the average difference in final writing performance between groups assigned to an educational intervention and comparison condition.
This differs from estimating the effect among participants who fully complied with the intervention.
The distinction matters because different analytical populations and handling strategies can answer different questions.
Although the formal ICH framework was developed for clinical trials, the broader principle of defining the target of inference is useful in other quantitative studies.
How Should Analysis Populations Be Specified?
The SAP should identify which observations contribute to each important analysis.
For randomized trials, this may involve an intention-to-treat-oriented analysis based on assigned groups, together with appropriately defined supplementary analyses.
However, the term intention-to-treat should not be used as a substitute for explaining how missing outcomes, exclusions, and intercurrent events will be handled.
Observational studies should specify eligibility, observation periods, and inclusion or exclusion rules relevant to the analytical sample.
Exclusions based on data quality or implausible values should follow defensible criteria rather than the direction of the estimated effect.
When different analyses use different populations, the reasons should be explicit.
How Detailed Should Variable Coding Be?
The plan should specify consequential variable transformations and coding decisions.
These may include reference categories, continuous versus categorical treatment, composite-score construction, centering, transformations, and the handling of special values.
For example, if age is included as a covariate, the plan should indicate whether it is modeled continuously, categorized using justified thresholds, or represented through a flexible functional form.
Arbitrary categorization can discard information and alter interpretation.
Not every software-specific formatting decision must be predetermined, but choices that affect estimates or hypothesis tests should be sufficiently clear.
A data dictionary can complement the SAP by documenting variables and permitted values.
Must the Exact Statistical Model Be Specified?
For a confirmatory primary analysis, the model should ordinarily be specified in enough detail to reproduce the intended estimate.
This may include the model family, link function, dependent variable, predictors, adjustment variables, interactions, covariance structure, and method for estimating uncertainty.
For example, a linear model comparing final writing scores might include treatment assignment and baseline writing performance as predictors.
If participants are nested within classrooms, the plan should address clustering through an appropriate design-based or model-based approach.
Writing only that a t-test, ANOVA, or regression will be performed may leave consequential choices unresolved.
However, some model details may legitimately depend on prespecified diagnostic rules or features that cannot be known before data collection.
Should Covariates Be Selected Before the Data Are Examined?
For confirmatory analyses, important adjustment variables should generally be specified prospectively using substantive knowledge and the research design.
In randomized trials, baseline covariate adjustment may improve precision when appropriately planned.
In observational causal studies, covariate selection should reflect a defensible confounding strategy rather than statistical significance alone.
Choosing covariates because they produce a preferred treatment estimate can undermine analytical credibility.
Researchers may conduct additional exploratory models, but these should be distinguished from the primary prespecified analysis.
How Should Missing Data Be Addressed?
The SAP should identify anticipated missing-data patterns and the primary strategy for handling missing observations.
Possible approaches include appropriately justified complete-case analysis, multiple imputation, likelihood-based methods, or other procedures suited to the design.
However, no method is universally appropriate.
Researchers should consider the missingness mechanism, the variables available, the analysis model, and the assumptions required.
For example, multiple imputation does not automatically remove bias when missingness depends on unobserved values after accounting for observed information.
Where missing outcomes could materially affect conclusions, planned sensitivity analyses should examine departures from the primary assumptions.
Detailed consideration of missing-data handling before data collection is especially important in longitudinal and intervention studies.
Should Assumption Checks Be Prespecified?
Yes, when the interpretation of the primary analysis depends on assumptions that can be assessed using the observed data.
The plan may specify residual diagnostics, checks of functional form, influential observations, or other relevant procedures.
However, not every assumption is directly testable. For example, the absence of unmeasured confounding cannot be established through a routine regression diagnostic.
Researchers should also avoid automatically switching statistical tests whenever a preliminary assumption test produces a particular p-value.
Where alternative methods are anticipated, the plan should explain the circumstances under which they will be used and how the resulting analyses will be interpreted.
How Should Statistical Significance and Uncertainty Be Specified?
For confirmatory hypothesis testing, the SAP should ordinarily state the significance level, whether tests are one-sided or two-sided, and how multiple comparisons will be addressed where relevant.
It should also specify appropriate effect estimates and uncertainty measures, such as confidence intervals.
A conventional significance threshold is not automatically appropriate for every study.
Researchers should justify the inferential approach according to the design, number of hypotheses, and consequences of statistical errors.
Statistical significance should not be treated as equivalent to practical importance or causal validity.
For descriptive or estimation-focused studies, interval estimates and substantive interpretation may be more central than dichotomous hypothesis tests.
How Detailed Should Secondary Analyses Be?
Secondary analyses should be planned sufficiently to distinguish prespecified objectives from later exploratory investigations.
However, the level of detail may vary according to their importance.
A key secondary outcome intended to support a formal claim may require substantial prespecification, including multiplicity considerations.
A supplementary descriptive analysis may require less extensive specification.
Researchers should avoid presenting secondary analyses as confirmatory merely because their results appear statistically significant.
The appropriate approach to prespecifying secondary analyses depends on the study's objectives and inferential claims.
Should Subgroup Analyses Be Included in the SAP?
Subgroup analyses should be specified prospectively when they are intended to investigate important differences in effects or associations.
The plan should identify the subgroup variable, categories or functional form, analytical comparison, and whether an interaction will be estimated.
For example, researchers may investigate whether an educational intervention's effect differs according to baseline digital access.
Comparing a significant result in one subgroup with a nonsignificant result in another does not establish that the effects differ.
An appropriate interaction analysis is generally needed to evaluate such a difference.
Subgroup analyses may have limited precision, particularly when the study was not designed or powered for them.
These considerations should inform subgroup analysis planning.
How Should Sensitivity Analyses Be Planned?
Sensitivity analyses examine how conclusions change under alternative assumptions or defensible analytical choices.
For example, a longitudinal study might examine whether results remain similar under different assumptions about missing outcomes.
The SAP should identify important assumptions and the alternative analyses intended to evaluate them.
However, sensitivity analyses should not be selected merely to find a version of the results that supports a preferred conclusion.
Researchers should explain what each analysis tests and how disagreement between results would affect interpretation.
Planning sensitivity analyses during research design can reveal information that must be collected to make them feasible.
What About Clustering and Repeated Measurements?
Clustering and repeated measurements should be addressed before data collection because they affect both analysis and sample-size planning.
For example, students within the same classroom may have correlated outcomes, while repeated scores from the same student are generally not independent.
The SAP should specify an appropriate analytical approach, such as a multilevel model, marginal model, or another justified method.
It should also consider the number of clusters, cluster sizes, measurement occasions, and relevant covariance assumptions.
Ignoring dependence can produce misleading standard errors and confidence intervals.
These issues should be coordinated with sample-size planning for clustering and repeated measures.
Must the SAP Name the Statistical Software?
Identifying the intended software and relevant procedures can improve reproducibility, particularly when different implementations may produce different results.
However, software names alone do not constitute an adequate analysis plan.
The scientific method should be described independently of a particular menu command.
For complex analyses, the plan may specify the intended estimation algorithm, relevant options, convergence criteria, and procedures for handling computational failures.
Version information and analysis code can be documented when the analyses are implemented.
Researchers should avoid allowing software availability to determine the research question or justify an inappropriate method.
What Details Can Legitimately Remain Flexible?
Some analytical decisions may depend on information that is unavailable before data collection.
Examples include the investigation of unexpected data errors, unforeseen convergence problems, or exploratory analyses prompted by new observations.
Flexibility is more defensible when the SAP specifies how such situations will be evaluated and how changes will be documented.
For confirmatory studies, researchers should avoid using knowledge of treatment effects or hypothesis-test results to select the primary model opportunistically.
Where possible, changes can be evaluated using blinded or otherwise appropriately restricted information.
The distinction between planned and exploratory analysis should remain visible in the final report.
When Should the SAP Be Finalized?
Major analytical requirements should be considered during study design, before recruitment and data collection create irreversible constraints.
In some trials, a detailed SAP is finalized after recruitment begins but before database lock, unblinding, or access to comparative outcome results, subject to applicable requirements.
This does not mean that researchers should postpone planning the primary analysis until the end of recruitment.
The protocol should already establish the essential design and analytical principles needed for sample-size planning and valid implementation.
The appropriate timing for specifying the primary analysis before recruitment depends on the study and its governance requirements.
How Should Changes to an SAP Be Documented?
Changes should be recorded with their dates, rationale, authorizations, and relationship to the information available when the decision was made.
For example, a change made before any outcome data are examined differs from a change selected after observing a treatment effect.
Researchers should preserve earlier versions rather than silently replace them.
Consequential deviations from prespecified analyses should be explained in the final report, with the original and revised analyses presented where appropriate.
Transparency does not automatically make every change scientifically justified, but it allows readers to evaluate its implications.
Can an SAP Reveal That the Research Design Is Inadequate?
Yes. Translating a research question into a concrete analytical procedure may reveal that the planned data cannot support the intended inference.
For example, a study may propose estimating institutional differences while including too few institutions to support the intended model or precision.
A study may also lack essential confounder measurements or appropriate follow-up timing.
These problems should be addressed through design revision rather than concealed by choosing a convenient statistical test after data collection.
Researchers should use analytical planning to assess whether the proposed study can answer its question before committing to implementation.
Watch Out
A statement such as "data will be analyzed using appropriate descriptive and inferential statistics" is not a sufficient confirmatory analysis plan. It leaves the outcome, comparison, model, adjustment variables, uncertainty, and potentially consequential analytical choices unspecified.
