03 · What You Need to Know
Developing Statistical, Qualitative, and Mixed-Methods Analysis Plans
What Is a Research Data Analysis Plan?
A data analysis plan explains how research evidence will be organized, examined, and interpreted to address the study's questions or objectives.
It establishes the relationship between the research design, the information collected, and the conclusions the researchers intend to draw.
For example, a study evaluating an AI-supported learning intervention may require an analysis that accounts for baseline achievement, the intervention comparison, and the way students were assigned or sampled.
A qualitative study exploring teachers' experiences with generative AI may instead require an interpretive approach capable of examining meanings, practices, and contextual relationships.
The analysis plan should therefore follow the methodological purpose of the study rather than begin with a preferred statistical test or software package.
Why Should Analysis Be Planned Before Data Collection?
Analysis planning can reveal whether the proposed data are sufficient and suitable for the research questions.
For example, a longitudinal analysis requires appropriately linked observations across measurement occasions. A study examining change cannot reliably estimate individual trajectories if the data collection system does not preserve the necessary identifiers.
Similarly, a qualitative study seeking to understand institutional differences may need sampling and contextual documentation that allow those differences to be interpreted.
Planning can also reduce opportunities to select analytical methods solely because they produce favorable findings.
However, prospective planning does not guarantee methodological validity. An inappropriate analysis remains inappropriate even when specified in advance.
What Is the Difference Between an Analysis Plan and an Analysis Procedure?
Analysis plan
Establishes the analytical objectives, methodological approach, principal decisions, and intended interpretation.
Analysis procedure
Describes how the planned analysis will be implemented through specific operational steps.
Analysis code or working record
Documents the actual computations, transformations, coding decisions, or other analytical activities performed.
For example, the analysis plan may specify a regression model and the variables it will include. The analysis code implements that model in a particular software environment.
Likewise, a qualitative plan may identify reflexive thematic analysis, while analytical memos document how interpretations develop through engagement with the data.
What Should a Quantitative Statistical Analysis Plan Include?
The necessary components depend on the study design, but confirmatory quantitative research commonly benefits from relatively detailed prospective specification.
| Component |
What to specify |
Why it matters |
| Research objectives |
Questions or hypotheses addressed by each principal analysis. |
Connects analysis with intended inference. |
| Analysis population |
Which participants or observations contribute to each analysis. |
Clarifies the population represented by estimates. |
| Primary outcome |
Variable definition, measurement occasion, and scoring. |
Identifies the central evidence. |
| Estimand |
The effect or quantity the analysis intends to estimate. |
Clarifies the scientific target. |
| Statistical model |
Principal model, variables, and relevant structure. |
Establishes the inferential approach. |
| Covariates |
Planned adjustments and their rationale. |
Reduces arbitrary analytical choices. |
| Missing data |
Primary strategy, assumptions, and relevant sensitivity analyses. |
Addresses incomplete observations. |
| Multiplicity |
Approach to multiple confirmatory comparisons where relevant. |
Supports appropriate error interpretation. |
| Sensitivity analyses |
Planned assessments of important assumptions or analytical choices. |
Evaluates robustness. |
| Reporting |
Estimates, uncertainty intervals, and relevant diagnostic information. |
Supports transparent interpretation. |
Not every item requires equal detail in every study. A descriptive survey, randomized trial, and complex longitudinal observational study have different analytical requirements.
Should the Primary Outcome Be Identified Before Analysis?
For confirmatory quantitative research, the primary outcome should generally be specified before relevant outcome information is examined.
The plan should define the variable, measurement instrument, scoring procedure, and relevant assessment time.
For example, stating that the study will examine academic performance is less precise than specifying the post-intervention achievement score on a defined assessment.
Where several outcomes are investigated, researchers should distinguish primary and secondary analyses and consider the implications of multiple comparisons.
Further guidance is available on preregistering the primary study outcome.
What Is an Estimand, and Why Does It Matter?
An estimand describes the quantity or treatment effect that a statistical analysis intends to estimate.
In intervention research, this may require specifying the target population, outcome, treatment comparison, handling of relevant intercurrent events, and population-level summary.
For example, a study may aim to estimate the effect of assigning students to an AI-supported learning intervention, regardless of whether every student uses the system as intended.
That question differs from estimating an effect among students who actually engage with the intervention at a specified level.
These analyses may require different assumptions and methods.
The ICH E9(R1) estimand framework provides formal guidance in clinical trials, while the broader principle of aligning the scientific question with the analytical target is relevant elsewhere.
Should the Statistical Model Be Prespecified?
Prespecification is particularly useful for principal confirmatory analyses.
The plan may identify the model family, outcome, explanatory variables, covariates, treatment of clustering, and relevant interactions.
For example, if students are randomized by classroom, the analysis should account appropriately for the cluster structure rather than automatically treat all observations as independent.
However, a model should not be selected merely because it is familiar or commonly used.
Its suitability depends on the research design, measurement scale, estimand, and assumptions.
Unexpected data characteristics may justify changes, but consequential departures should be explained and distinguished from the original plan.
How Should Missing Data Be Addressed?
The analysis plan should consider the possibility that some observations will be incomplete.
For example, students may miss follow-up assessments, withdraw from the study, or submit incomplete questionnaires.
Different missing-data mechanisms have different implications for estimation and interpretation.
Complete-case analysis, multiple imputation, likelihood-based methods, and other approaches rely on assumptions that should be evaluated in relation to the design.
There is no universally appropriate method for every dataset.
Researchers should distinguish procedures for preventing avoidable missingness from analytical methods for addressing the missing information that remains.
Should Data Exclusion Rules Be Planned?
Consequential exclusion decisions should generally be established before examining relevant outcomes in confirmatory research.
These may concern duplicate records, invalid measurements, ineligible participants, or observations that fail predefined quality requirements.
However, eligibility exclusions, measurement exclusions, and decisions about the analysis population are not interchangeable.
For example, excluding randomized participants because they did not complete the intervention may change the scientific question being estimated.
Researchers should therefore justify exclusion rules according to the study design and intended estimand.
The principles for preregistering data exclusion rules are relevant when these decisions affect confirmatory claims.
Should Subgroup Analyses Be Prespecified?
Planned subgroup analyses should identify the relevant groups, comparisons, and statistical approach.
For example, researchers may investigate whether an intervention effect differs according to prior academic achievement.
However, observing a statistically significant effect in one subgroup and a nonsignificant effect in another does not by itself establish that the effects differ.
Appropriate interaction analysis may be necessary.
Exploratory subgroup analyses can still be informative, but their interpretation should reflect the increased risk of chance findings and the absence of prospective specification.
What Should a Qualitative Analysis Plan Include?
A qualitative analysis plan should identify the analytical orientation and explain how the researcher intends to engage with the evidence.
Depending on the methodology, relevant elements may include theoretical assumptions, data preparation, coding or interpretation practices, reflexivity, methodological documentation, and the relationship between data generation and analysis.
For example, a study using reflexive thematic analysis should explain its interpretive orientation and how patterns of meaning will be developed.
A framework analysis may involve a more structured analytical matrix, while grounded theory may integrate analysis with theoretical sampling.
These approaches should not be treated as interchangeable techniques merely because they involve qualitative data.
Must Qualitative Coding Categories Be Fixed Before Collection?
No universal requirement applies.
Some qualitative content analysis approaches use deductive categories informed by prior theory or an established framework.
Other approaches develop categories or interpretations through engagement with the data.
For example, reflexive thematic analysis does not ordinarily require a fixed codebook or inter-coder agreement as evidence of analytical quality.
Researchers should explain whether their approach is primarily deductive, inductive, or combines different analytical orientations.
The degree of prespecification should remain consistent with the methodology rather than imitate confirmatory statistical analysis.
How Should Reflexivity Be Addressed?
Reflexivity may be important when the researcher's assumptions, positioning, and relationships influence data generation and interpretation.
For example, a university teacher interviewing colleagues about generative AI may need to consider how professional relationships shape the interview interaction.
Reflexive practices may include methodological memos, critical examination of assumptions, and discussion of consequential analytical decisions.
However, reflexivity is not a mechanical procedure that automatically removes bias.
Its value depends on how meaningfully it informs the conduct and interpretation of the research.
Should Qualitative Researchers Specify How Themes Will Be Developed?
Yes, at the level appropriate to the chosen analytical approach.
For example, a reflexive thematic analysis plan may describe familiarization, coding, theme development, refinement, and interpretive writing without predicting the themes that will emerge.
However, researchers should avoid treating themes as objects that necessarily exist in the data independently of analytical interpretation.
Different qualitative traditions conceptualize coding, categorization, and theme development differently.
The analysis plan should therefore explain the intended approach rather than promise that a particular number of themes will be discovered.
How Should Qualitative Analytical Changes Be Documented?
Qualitative analysis may develop iteratively as researchers encounter new evidence and refine their understanding.
Consequential decisions can be documented through analytical memos, revised interview guides, sampling records, and other methodological documentation.
For example, a researcher may record why emerging accounts of institutional responsibility prompted additional interviews or a revised analytical focus.
Such changes may be methodologically appropriate, but they should remain consistent with the research purpose and applicable ethics requirements.
Transparent documentation supports evaluation without imposing artificial rigidity.
What Should a Mixed-Methods Analysis Plan Include?
A mixed-methods analysis plan should explain both component analyses and how their findings will be integrated.
Integration may occur through sampling, data collection, analysis, or interpretation.
For example, quantitative achievement results may identify patterns that inform subsequent interviews with students or instructors.
The protocol should explain how the qualitative component will help interpret those patterns and how integrated conclusions will be developed.
Merely presenting quantitative results followed by interview quotations does not necessarily constitute meaningful integration.
The analytical relationship between components should be justified by the mixed-methods design.
How Much Analytical Flexibility Is Appropriate?
The answer depends on the intended claims and methodological orientation.
Confirmatory quantitative studies generally require stronger prospective commitments because analytical choices made after observing results can affect inferential error rates and the credibility of hypothesis tests.
Exploratory research may legitimately compare alternative models, investigate unexpected relationships, or develop new hypotheses.
Qualitative research may require iterative interpretation and responsive analytical decisions.
However, flexibility should not be confused with retrospectively presenting an exploratory finding as a prespecified confirmatory result.
The principles for prespecifying consequential research decisions help distinguish fixed commitments from legitimate discretion.
Can the Analysis Plan Change After Data Collection Begins?
Yes, when changes are scientifically justified and appropriately documented.
For example, a planned statistical model may prove unsuitable because its assumptions are seriously violated.
Researchers may need to adopt a more appropriate analysis, but they should explain the reason and distinguish the revised analysis from the original plan.
The timing of the decision relative to access to relevant outcome information matters.
Changes made before examining outcomes may have different inferential implications from changes selected after reviewing favorable or unfavorable results.
Where the study is a Registered Report, consequential changes may also require editorial consultation under the journal's policies.
Should Exploratory Analyses Be Included in the Protocol?
They may be, particularly when the research team anticipates investigating additional patterns beyond the primary hypotheses.
However, exploratory analyses need not always be specified exhaustively before collection.
The important distinction is between analyses intended to test prospective claims and analyses used to generate or refine hypotheses.
For example, examining unexpected associations among student engagement indicators may be useful exploratory work.
Such findings should be interpreted cautiously and, where appropriate, evaluated in subsequent research.
How Detailed Should the Analysis Plan Be in the Main Protocol?
The main protocol should provide enough detail to establish the intended analytical strategy and permit scientific review.
Complex confirmatory studies may benefit from a separate statistical analysis plan containing model specifications, estimands, missing-data procedures, and planned sensitivity analyses.
Qualitative studies may use supporting methodological documents or analytical protocols where appropriate.
The main protocol should identify and reference these documents and remain consistent with their consequential commitments.
Researchers should not move essential analytical decisions into an informal document merely to avoid appropriate review.
Does Naming Statistical Software Constitute an Analysis Plan?
No. Identifying SPSS, R, Stata, Python, NVivo, ATLAS.ti, or another platform does not establish an analytical methodology.
Software provides tools for implementing analytical decisions. It does not independently justify those decisions.
For example, stating that data will be analyzed using R does not identify the primary outcome, statistical model, or treatment of missing data.
Similarly, stating that interview transcripts will be analyzed using qualitative software does not explain the interpretive approach.
The plan should establish the analytical logic before discussing software implementation.
How Can Researchers Check Whether the Analysis Plan Is Adequate?
A useful review traces each research question through the required data, analytical procedure, and intended interpretation.
For example, if a study asks whether an intervention improves achievement, the plan should establish what improvement means, which comparison will be made, and how the design supports that comparison.
If a qualitative study asks how teachers negotiate responsibility for AI-generated materials, the plan should explain how the selected analytical approach can examine those experiences and interpretations.
Researchers should also evaluate whether the proposed sample, instruments, and data structure support the analysis.
Where important decisions remain unclear, the plan may need further development before collection begins.
Does a Detailed Analysis Plan Guarantee Credible Findings?
No. A detailed plan can improve transparency and reduce some forms of analytical discretion, but it cannot compensate for invalid measurements, inappropriate sampling, unaddressed confounding, or unsuitable assumptions.
For example, a perfectly prespecified regression model cannot establish a causal effect when the design and assumptions do not support causal inference.
Similarly, an elaborate qualitative coding procedure does not guarantee a persuasive interpretation if the approach is inconsistent with the research question or theoretical orientation.
The analysis plan should be evaluated as part of the complete research design.
Watch Out
Do not treat a list of statistical tests or a generic promise to code qualitative data as a complete analysis plan. The plan should explain how the selected approach answers the research questions, what assumptions it requires, and which consequential decisions are fixed or legitimately flexible. Prespecification improves transparency but does not substitute for methodological justification.
