03 · What You Need to Know
Planning Secondary Analyses Without Confusing Confirmation and Exploration
What Is a Secondary Analysis?
A secondary analysis is an analysis that addresses an objective other than the study's designated primary objective. It may investigate additional outcomes, relationships, mechanisms, or descriptive characteristics using data collected within the study.
For example, a randomized educational intervention may primarily estimate an effect on writing performance and secondarily examine feedback satisfaction.
The secondary analysis may be prespecified, exploratory, descriptive, or intended to support a formal inferential claim. Its designation as secondary does not, by itself, determine which of these applies.
The term should also be distinguished from secondary data analysis, which commonly refers to analyzing data originally collected for another purpose. A planned secondary outcome analysis can use newly collected primary study data.
How Do Primary, Secondary, and Exploratory Analyses Differ?
| Analysis category |
Principal role |
Planning implication |
| Primary analysis |
Addresses the principal research objective and intended main inference. |
Requires substantial prospective specification for confirmatory research. |
| Prespecified secondary analysis |
Addresses an additional objective identified before relevant results are examined. |
Requires detail proportionate to its intended inferential role. |
| Exploratory analysis |
Investigates patterns, relationships, or hypotheses without a fully established confirmatory framework. |
Allows flexibility with transparent reporting and cautious inference. |
| Post hoc analysis |
Is developed after a specified decision point, often after relevant data or results have been examined. |
Requires disclosure of timing, rationale, and analytical limitations. |
These categories are not entirely mutually exclusive. An exploratory analysis can be planned prospectively, and a secondary analysis can be developed post hoc.
Researchers should therefore describe both the analysis's scientific purpose and when its methods were established.
Why Should Important Secondary Analyses Be Planned Early?
Early planning helps ensure that the study collects the information required for secondary objectives.
For example, a research team interested in the relationship between intervention engagement and academic performance may need to collect engagement indicators at specified times.
If the objective emerges after data collection, essential measurements may be missing or unsuitable.
Prospective planning can also reduce opportunities to select analyses because their results are statistically significant or favorable.
However, prespecification cannot correct an inappropriate design or compensate for inadequate measurement.
The scientific value of an analysis depends on its question, methods, assumptions, and evidence, not solely on when it was documented.
Must Every Secondary Research Question Have a Planned Analysis?
Substantive secondary questions included in a confirmatory research protocol should generally have corresponding analytical strategies.
For example, if the protocol states that the study will compare feedback satisfaction between intervention conditions, the researchers should identify the satisfaction measure, comparison, and intended analysis.
However, a broad exploratory objective may reasonably allow several analytical approaches to emerge during investigation.
The appropriate level of planning depends on whether the study intends to make a specific inferential claim or generate hypotheses.
Researchers should avoid presenting numerous secondary objectives without considering whether the data and resources can adequately address them.
How Detailed Should Secondary Analyses Be?
The necessary detail depends on the analysis's scientific importance and intended interpretation.
A key secondary endpoint intended to support a formal claim may require a defined estimand, analytical population, model, covariates, missing-data strategy, and multiplicity procedure.
A supplementary descriptive analysis may require only clearly defined variables, summary statistics, and uncertainty measures.
For example, reporting the proportion of students who completed an intervention may require less elaborate modeling than estimating a secondary treatment effect on psychological well-being.
The plan should be sufficiently specific that consequential choices cannot easily be made according to which results appear most favorable.
Should Secondary Outcomes Be Defined Before Data Collection?
Yes, when they form part of the study's planned objectives.
The plan should specify the construct, measurement instrument, scoring method, and relevant assessment time.
For example, student engagement could refer to platform activity, self-reported behavioral engagement, or another construct. These are not interchangeable simply because they share a general label.
Researchers should also distinguish secondary outcomes from implementation indicators, mediators, and potential confounders.
The same variable can have different analytical roles in different research questions, but those roles should be explained.
Can Secondary Analyses Be Confirmatory?
Potentially. A secondary analysis may support a confirmatory claim when its hypothesis, methods, and inferential framework have been appropriately specified and the study provides adequate evidence.
However, simply identifying an outcome before data collection does not automatically make the analysis confirmatory.
Researchers must consider statistical power, multiplicity, measurement validity, and whether the study's design supports the intended conclusion.
In clinical trials with several formal endpoints, hierarchical testing or other procedures may be used to control the relevant familywise error rate.
Such procedures should be selected according to the inferential objectives rather than adopted mechanically.
How Does Multiple Testing Affect Secondary Analyses?
Testing several hypotheses increases the opportunity to observe apparently significant results by chance when the tests are interpreted without an appropriate multiplicity framework.
For example, conducting many independent tests at a nominal significance level of 0.05 produces a greater probability of at least one false positive under the global null hypothesis.
However, tests within a study are often correlated, and the appropriate adjustment depends on the family of hypotheses and intended claims.
Possible approaches include familywise error control, false discovery rate procedures, hierarchical testing, or explicitly exploratory interpretation.
Not every descriptive secondary analysis requires a formal multiplicity correction.
The important issue is whether researchers intend to make confirmatory claims across multiple comparisons.
Should Secondary Analyses Be Included in Sample-Size Planning?
They should be considered when the study intends to draw important conclusions from them.
A sample size adequate for the primary outcome may be insufficient for secondary outcomes with different variability, event frequencies, or expected effect sizes.
For example, a study powered to detect a difference in writing scores may have limited precision for detecting differences in a relatively uncommon adverse experience.
Researchers should distinguish analyses that the study is designed to support from those intended to provide preliminary estimates.
It may be impractical to power a study for every secondary objective. In such cases, the limitations should be anticipated and reported.
Can Secondary Analyses Use Different Statistical Models?
Yes. Different outcomes and research questions may require different analytical approaches.
For example, a continuous writing score may be analyzed using an appropriate linear model, while a binary outcome concerning completion may require a model suitable for binary data.
Repeated engagement measurements may require methods accounting for within-participant dependence.
Researchers should choose methods according to outcome properties, design, and estimand rather than apply the primary model indiscriminately.
The analysis plan should explain important differences in models and the conclusions they support.
Should Secondary Analyses Use the Same Analytical Population?
Not necessarily. Different secondary outcomes may be defined for different populations or require different observation periods.
However, changes in analytical population can alter the scientific question and introduce selection bias.
For example, an analysis restricted to students who completed every intervention activity may no longer estimate the effect of randomized assignment among all eligible randomized students.
Researchers should specify the population relevant to each analysis and distinguish descriptive analyses of participants' experiences from causal comparisons.
Exclusion rules should be justified rather than selected according to the direction of the results.
How Should Missing Data Be Handled in Secondary Analyses?
Secondary outcomes may have different missing-data patterns from the primary outcome.
For example, final writing assessments may be nearly complete while optional satisfaction questionnaires have substantial nonresponse.
The primary outcome's missing-data strategy may not be suitable for every secondary measure.
Researchers should specify important assumptions and methods for secondary outcomes, particularly when the analyses support substantive claims.
Complete-case analysis, multiple imputation, and likelihood-based approaches each require assumptions that should be evaluated.
Missingness related to unobserved outcomes may require sensitivity analyses or more cautious interpretation.
Can Secondary Analyses Investigate Mechanisms?
Yes, but claims about mechanisms require particular care.
For example, researchers may hypothesize that AI-assisted feedback improves writing performance by increasing revision activity.
An association between revision activity and writing performance does not, by itself, establish mediation.
Formal mediation analysis may require assumptions about confounding, temporal ordering, and the relationships among treatment, mediator, and outcome.
Random assignment of the intervention does not automatically eliminate confounding of the mediator-outcome relationship.
Mechanistic analyses should therefore be distinguished from descriptive associations and interpreted according to their assumptions.
How Do Secondary Analyses Differ From Subgroup Analyses?
A secondary analysis may investigate an additional outcome or relationship, while a subgroup analysis examines results within or across defined subsets of a population.
A subgroup analysis can itself be secondary, but the concepts describe different dimensions of the analysis.
For example, comparing intervention effects on satisfaction is a secondary outcome analysis. Examining whether the effect on writing performance differs according to baseline digital access is a subgroup or effect-modification analysis.
Subgroup analyses require attention to interaction tests, precision, and multiplicity.
Researchers should consider planning subgroup analyses before data collection separately from the broader secondary analysis strategy.
What If an Interesting Secondary Question Emerges After Data Collection?
Researchers may investigate it when the available data and research design support a meaningful analysis.
Exploratory work is an important part of scientific inquiry and can generate hypotheses for subsequent studies.
However, an analysis developed after inspecting results should not be described as though its hypothesis and methods were prespecified.
Researchers should document the rationale, analytical decisions, number or scope of relevant comparisons, and limitations.
Where findings are particularly consequential, independent replication or confirmation may be warranted.
Can Researchers Change a Planned Secondary Analysis?
Yes, when there is a defensible reason.
For example, an instrument may prove unusable, a planned model may fail to converge, or a variable may be measured differently from what the protocol anticipated.
However, changes made after examining relevant results are more vulnerable to selective analysis concerns.
Researchers should preserve earlier versions of the plan, document the reasons and timing of changes, and distinguish revised analyses from those originally specified.
Consequential changes may require protocol amendments or registration updates under applicable requirements.
Should Secondary Analyses Be Reported If They Are Not Statistically Significant?
Prespecified secondary analyses should generally be reported transparently regardless of whether their results cross a significance threshold.
Selective reporting of only favorable or statistically significant outcomes can misrepresent the evidence.
However, the appropriate reporting format depends on the study, the number of analyses, and relevant publication requirements.
Supplementary materials may be useful for extensive results, provided important findings and limitations remain accessible.
Effect estimates and uncertainty should be reported where appropriate rather than reducing interpretation to significant versus nonsignificant labels.
How Should Secondary Findings Be Interpreted When the Primary Result Is Null?
A null primary result does not automatically invalidate every secondary finding.
However, the interpretation depends on the study's prespecified inferential hierarchy, multiplicity strategy, and scientific objectives.
For example, a trial may find little evidence of improved writing performance while identifying favorable differences in student satisfaction.
The satisfaction result may still be informative, but it does not establish that the intervention achieved its primary educational objective.
Where formal hierarchical testing prevents confirmatory claims about later endpoints, those findings should be interpreted accordingly.
Researchers should avoid promoting a favorable secondary outcome into the primary conclusion after observing the results.
How Should Secondary Analyses Be Documented in the SAP?
The SAP should identify the analysis objective, outcome or variables, target population, intended comparison, statistical method, and uncertainty measures for each important secondary analysis.
It should also address missing-data handling, multiplicity, and sensitivity analyses where relevant.
Researchers may organize secondary analyses by scientific priority rather than provide an undifferentiated list of every possible statistical test.
The plan should distinguish prespecified secondary analyses from analyses explicitly designated as exploratory.
The appropriate level of statistical analysis plan detail should reflect the importance and intended claims of each analysis.
How Should Secondary Analyses Be Reported?
Reports should identify which analyses were prespecified, which were modified, and which were developed after relevant results were examined.
Researchers should provide appropriate effect estimates, uncertainty measures, and discussion of multiplicity and statistical power.
Design-specific reporting standards, such as CONSORT for randomized trials and STROBE for observational studies, may provide relevant guidance.
Transparent reporting should distinguish evidence supporting a planned secondary hypothesis from findings that primarily generate questions for further investigation.
The objective is not to diminish secondary findings, but to represent their evidentiary status accurately.
Watch Out
Do not treat every prespecified secondary analysis as confirmatory or every post hoc analysis as scientifically worthless. Prespecification, multiplicity control, statistical precision, and the study's inferential design all affect what a secondary result can legitimately establish.
