03 · What You Need to Know
How Statistical Power Should Influence Secondary Outcome Planning
What Is a Secondary Outcome?
A secondary outcome measures an additional scientifically relevant effect, consequence, or characteristic beyond the study's principal outcome.
For example, a trial evaluating an educational intervention may designate achievement as primary and engagement as secondary.
Secondary status does not mean that the outcome is unimportant.
It indicates a particular role within the study's objective and analytical hierarchy.
Some secondary outcomes support formal confirmatory claims. Others provide supportive evidence, help explain mechanisms, or generate hypotheses for subsequent research.
The distinction is essential when deciding whether individual statistical power is required.
What Is Statistical Power?
Statistical power is the probability that a specified testing procedure will reject a null hypothesis under a particular alternative hypothesis.
For example, 80% power means that the procedure has an 80% probability of rejecting the null hypothesis under the assumed effect size and other planning conditions.
Power depends on the effect of interest, sample size, variability, significance criterion, statistical model, and design structure.
It is not a fixed property of a dataset that applies equally to every outcome.
Nor does 80% power mean that an observed significant result has an 80% probability of being correct.
Why Can Secondary Outcomes Have Different Power?
Different outcomes may have different distributions and expected treatment effects.
For example, achievement scores may be approximately continuous, while course completion is binary.
A study may have sufficient power to detect a meaningful difference in achievement but inadequate power to detect a small difference in completion rates.
Measurement reliability, missing data, baseline variability, and clustering may also differ across outcomes.
Consequently, a sample size calculation based on one outcome cannot automatically establish adequate power for another.
What Is a Confirmatory Secondary Outcome?
A confirmatory secondary outcome supports a prespecified statistical claim beyond the primary objective.
For example, a clinical trial may establish a testing hierarchy in which a key secondary outcome can support a formal efficacy claim after the primary hypothesis meets its criterion.
Such outcomes require appropriate consideration of power, multiplicity, and the relevant success criterion.
Researchers should not assume that calling an outcome secondary removes the statistical requirements associated with a confirmatory claim.
The intended inferential role is more important than the label alone.
What Is an Exploratory Secondary Outcome?
An exploratory secondary outcome is investigated to characterize additional effects, generate hypotheses, or inform future research.
It may be scientifically valuable even when the study is not powered to detect a particular effect on that outcome.
For example, a study primarily evaluating achievement may examine changes in motivation to identify potentially useful patterns.
However, exploratory findings should not be presented as definitive evidence merely because a nominal p-value falls below 0.05.
Researchers should report effect estimates, uncertainty, and the exploratory status of the analysis.
When Should a Secondary Outcome Be Individually Powered?
Individual or coordinated power planning becomes important when the study intends to make a consequential confirmatory claim about the secondary outcome.
For example, a treatment may be expected to improve the primary outcome and also demonstrate a clinically important quality-of-life benefit.
If the latter claim is essential to the study's objectives, researchers should evaluate whether the design provides sufficient power for it.
However, the correct calculation may involve the joint testing procedure rather than a separate standalone power analysis.
Hierarchical testing, correlations among outcomes, and the definition of study success can affect the required sample size.
What Is the Difference Between Individual and Joint Power?
Individual power concerns the probability of rejecting one specified null hypothesis under an assumed alternative.
Joint power concerns the probability of satisfying a combined success criterion involving several hypotheses.
For example, a study may require significant results on both achievement and self-regulation.
Even if each test has 80% power, the probability that both succeed may be lower than 80%.
Correlated outcomes require an appropriate joint calculation. Researchers should evaluate the complete decision framework rather than assume that adequate individual power guarantees adequate overall power.
How Does Multiplicity Affect Secondary Outcome Power?
When several secondary hypotheses can support confirmatory claims, the study may require a multiplicity procedure.
Such procedures can make individual rejection thresholds more stringent, reducing power at a fixed sample size.
For example, a Bonferroni-adjusted threshold may be smaller than the conventional 0.05 threshold.
However, Bonferroni is not universally required. Hierarchical testing and other procedures may provide appropriate control while reflecting scientific priorities.
The principles of multiplicity planning for multiple outcomes should therefore be incorporated into secondary outcome power calculations.
Can Hierarchical Testing Protect Confirmatory Secondary Claims?
Yes. A prespecified hierarchical procedure may permit formal testing of a secondary hypothesis after the primary hypothesis meets its criterion.
For example, achievement may be tested first, followed by engagement.
Under an appropriate fixed-sequence procedure, the secondary hypothesis can be tested confirmatorily only when the preceding testing condition is satisfied.
However, the secondary outcome must still have adequate statistical information if the researchers expect to detect a scientifically meaningful effect.
Multiplicity control and statistical power address different requirements.
A valid hierarchy does not guarantee that the secondary test is adequately powered.
Does a Nonsignificant Secondary Outcome Mean No Effect?
No. A nonsignificant result means that the analysis did not meet its specified rejection criterion.
It does not establish that the true effect is exactly zero.
When power is limited, the study may fail to detect effects that are scientifically meaningful.
Researchers should examine the estimated effect and confidence interval to determine which effect sizes remain compatible with the data.
For example, a wide interval spanning meaningful benefit and harm indicates substantial uncertainty.
It would be inappropriate to interpret such a result as convincing evidence of no effect.
Can a Significant Secondary Result Be Credible if the Study Was Not Powered for It?
Potentially. Lack of a dedicated power calculation does not automatically invalidate a statistical result.
However, its interpretation depends on the hypothesis specification, testing procedure, multiplicity, measurement quality, and broader study context.
An exploratory secondary finding may provide useful evidence even when the study was not designed to support a definitive claim.
Nevertheless, researchers should not retrospectively describe an exploratory result as confirmatory simply because it is statistically significant.
The question of significant secondary outcomes after a nonsignificant primary result requires particular caution.
Should Every Secondary Outcome Have a Separate Sample Size Calculation?
No. A separate calculation for every outcome may be unnecessary when the outcomes are supportive or exploratory.
However, researchers should understand the likely precision and detectable effects for important secondary measures.
For key confirmatory secondary outcomes, power may need to be evaluated under a coordinated testing strategy.
Researchers can sometimes use one sample size that satisfies several important requirements, but this must be demonstrated rather than assumed.
The largest individual sample size requirement does not necessarily guarantee adequate joint power for a multi-hypothesis success criterion.
How Does Outcome Variability Affect Power?
For continuous outcomes, greater unexplained variability generally reduces power for detecting a fixed mean difference at a given sample size.
Measurement error may also reduce the precision of estimated effects.
For example, a standardized achievement test may provide more precise information than an unreliable short questionnaire measuring motivation.
However, measurement reliability is only one consideration. The anticipated treatment effect and analysis model also matter.
Researchers should use relevant empirical evidence when estimating planning parameters rather than assuming that all outcomes have similar variability.
How Do Binary and Rare Outcomes Affect Power?
Binary outcomes depend on the event probabilities in the compared groups.
When events are uncommon, a study may observe relatively few events even with substantial enrollment.
For example, course withdrawal may occur infrequently in a short educational intervention study.
A trial adequately powered for achievement may therefore provide limited precision for withdrawal differences.
Researchers should avoid interpreting the absence of statistically significant differences in rare outcomes as evidence that the intervention has no effect on them.
How Does Clustering Affect Secondary Outcome Power?
In cluster-randomized studies, participants within the same classroom, school, clinic, or institution may have correlated outcomes.
This correlation reduces the amount of independent statistical information relative to an individually randomized design with the same total participant count.
Importantly, intracluster correlation can differ across outcomes.
For example, achievement may exhibit substantial classroom-level clustering, while an individual attitude measure may have a different correlation structure.
Researchers should therefore avoid applying one design effect indiscriminately to every secondary outcome.
Power calculations should reflect the actual randomization unit, cluster sizes, outcome characteristics, and analytical model.
Can Missing Data Affect Secondary Outcome Power?
Yes. Secondary outcomes may have different missing-data rates from the primary outcome.
For example, achievement scores may be available through institutional records, while motivation questionnaires may be completed by only a subset of participants.
Reduced outcome availability can lower precision and power.
Missing data may also introduce bias, depending on the missingness mechanism and analytical assumptions.
Researchers should therefore plan data collection and missing-data handling for important secondary outcomes rather than assume that the primary outcome's retention rate applies to every measure.
Should Secondary Outcomes Be Removed if They Are Underpowered?
Not automatically. An underpowered outcome may still provide useful descriptive or exploratory information.
For example, a small study may collect safety outcomes even when it cannot reliably detect uncommon adverse events.
Removing such measures could conceal relevant information.
However, researchers should evaluate whether each outcome contributes sufficient scientific value to justify measurement burden and analytical complexity.
The decision should reflect the number of outcomes the study can meaningfully support.
Can Researchers Calculate Power After Seeing the Results?
Retrospective or observed power calculated by substituting the observed effect into a conventional power formula is generally not a useful measure of evidential strength.
For a fixed test and sample size, observed power is largely a transformation of the observed test statistic or p-value.
It does not resolve the uncertainty about the true effect.
Researchers should instead examine effect estimates, confidence intervals, and the range of scientifically meaningful effects compatible with the data.
Prospective power calculations remain useful for planning future studies using justified effect-size assumptions.
What About Precision-Based Planning?
Not every secondary analysis needs to be framed around the probability of rejecting a null hypothesis.
Some outcomes are primarily intended to estimate an effect with useful precision.
For example, researchers may wish to estimate the difference in student satisfaction scores without making a definitive claim of superiority.
In such cases, planning around the expected width of a confidence interval may be more appropriate than conventional power for a significance test.
The target precision should reflect the scientific interpretation and intended use of the estimate.
How Should Secondary Outcome Results Be Reported?
Researchers should report secondary outcomes according to their prespecified roles, including nonsignificant findings.
Effect estimates and uncertainty intervals should accompany relevant statistical tests.
Where multiplicity procedures apply, the report should identify the adjusted testing framework.
Exploratory findings should be distinguished from confirmatory results.
Researchers should also disclose important limitations involving measurement reliability, missing data, and inadequate statistical information.
The goal is to communicate what the data support without overstating either positive or negative findings.
Watch Out
A study powered for its primary outcome is not automatically powered for its secondary outcomes. Conversely, an outcome without dedicated confirmatory power can still provide useful evidence. The inferential claims must match the purpose, precision, and statistical support of the analysis.
