03 · What You Need to Know
How Multiplicity Planning Depends on the Study's Inferential Objectives
What Is Multiplicity in Research?
Multiplicity arises when a research design provides multiple opportunities to make statistical claims, particularly claims involving rejection of null hypotheses.
These opportunities may arise from multiple outcomes, treatment comparisons, subgroups, analysis methods, or interim assessments.
For example, a study might test whether an intervention improves achievement, motivation, engagement, and retention.
If each outcome is treated as an independent opportunity to declare confirmatory success, the overall false-positive risk may exceed the nominal significance level of an individual test.
The consequences depend on the relationship among the hypotheses and the definition of study success.
Why Must Multiplicity Be Planned Before Data Collection?
Multiplicity planning establishes how the study will interpret several statistical results before researchers know which outcomes will appear favorable.
Without such planning, investigators may select the most significant outcome and present it as the central finding.
This creates a risk of selective inference.
Prospective planning also affects statistical power and sample size. A study using a more stringent testing threshold may require additional information to detect the intended effects.
The appropriate time to establish the strategy is during protocol and statistical analysis planning, before relevant results can influence the selection of hypotheses or procedures.
Does Every Study With Several Outcomes Need Adjustment?
No. The mere collection of multiple outcomes does not automatically establish a requirement for one universal adjustment.
The central question is which hypotheses support confirmatory claims and how those claims are related.
For example, a study may designate achievement as its sole primary confirmatory outcome while measuring motivation and satisfaction for exploratory purposes.
The primary test does not necessarily require adjustment merely because additional outcomes are measured.
However, if the researchers intend to claim confirmatory success whenever any of several outcomes is significant, the multiple testing opportunities must be addressed.
Researchers should distinguish collecting information from making multiple confirmatory claims.
What Is the Familywise Error Rate?
The familywise error rate, or FWER, is the probability of rejecting at least one true null hypothesis within a specified family of tests.
It is particularly relevant when any false-positive confirmatory claim would be consequential.
For example, a study might intend to establish efficacy on any one of four primary outcomes.
Testing each hypothesis at a nominal significance level of 0.05 without an appropriate strategy can increase the probability of at least one false rejection.
This calculation illustrates the potential accumulation of false-positive opportunities. It does not imply that every study measuring four outcomes has an 18.5% familywise error rate.
What Is Strong Familywise Error Control?
Strong FWER control requires maintaining the specified error limit under all relevant configurations of true and false null hypotheses.
Weak control addresses a more restricted configuration, commonly the global null in which all hypotheses in the family are true.
The distinction matters because a study may contain some outcomes with genuine effects and others with no effect.
A confirmatory procedure should provide the error protection required by the intended claims across the relevant configurations.
Researchers should not assume that demonstrating acceptable performance only when every null hypothesis is true establishes strong control.
How Do Different Primary Outcome Structures Affect Multiplicity?
| Outcome structure |
Success criterion |
Multiplicity consideration |
| Single primary outcome |
One specified primary hypothesis is rejected. |
No multiple-primary-outcome adjustment is ordinarily needed for that single hypothesis. |
| Co-primary outcomes |
All designated primary hypotheses must meet their criteria. |
An intersection-union structure may control the overall false-positive claim without conventional alpha splitting, but joint power requires attention. |
| Alternative primary outcomes |
Any one of several hypotheses may establish success. |
Appropriate multiplicity control is generally needed. |
| Hierarchical outcomes |
Hypotheses are tested in a specified sequence. |
Valid gatekeeping can control error while prioritizing claims. |
| Exploratory outcomes |
Results generate or refine hypotheses. |
Interpretation and multiplicity handling depend on exploratory objectives. |
The most important distinction is between requiring all primary outcomes to succeed and permitting success through any one of several outcomes.
These structures cannot be treated as interchangeable simply because both involve multiple measurements.
Why Are Co-Primary Outcomes Different?
Co-primary outcomes may be appropriate when the scientific claim requires success on every designated endpoint.
For example, a treatment may need to improve both symptoms and functional ability to satisfy its principal objective.
Under an intersection-union testing structure, the overall claim is established only when every component hypothesis is rejected.
When appropriately formulated, this structure can control the probability of a false overall success claim without dividing alpha across component tests.
However, requiring all outcomes to succeed may reduce the probability of overall study success.
Researchers should therefore evaluate joint power rather than assume that powering each outcome separately is sufficient.
What Happens When Any Primary Outcome Can Establish Success?
A different problem arises when researchers permit a positive conclusion if any one of several primary outcomes meets its significance criterion.
For example, an educational intervention might be declared effective if it improves achievement, engagement, motivation, or retention.
When several true null hypotheses are tested, this approach creates multiple opportunities for a false-positive claim.
Appropriate procedures may include Bonferroni adjustment, Holm testing, or methods that use the dependence structure among outcomes.
The chosen procedure should be consistent with the intended error criterion and statistical power requirements.
What Is the Bonferroni Procedure?
Bonferroni adjustment controls familywise error by allocating the significance level across a family of hypotheses.
Bonferroni is straightforward, but it may be conservative, particularly when outcomes are strongly correlated.
Its simplicity should not be confused with universal optimality.
What Is the Holm Procedure?
The Holm procedure is a step-down multiple-testing method that controls FWER.
It orders the p-values from smallest to largest and compares them with progressively less stringent thresholds.
For four hypotheses and FWER = 0.05, the ordered thresholds are 0.0125, approximately 0.0167, 0.025, and 0.05.
Testing stops when a hypothesis fails its applicable threshold, leaving that hypothesis and all subsequent hypotheses unrejected.
The procedure generally offers at least as much power as simple Bonferroni adjustment while retaining strong FWER control under arbitrary dependence when the p-values are valid.
However, it does not automatically account for every other design feature, such as repeated interim testing or adaptive hypothesis selection.
When Is Hierarchical Testing Appropriate?
Hierarchical testing evaluates hypotheses according to a prespecified order.
For example, a study might first test achievement, then engagement, then motivation.
Under a valid fixed-sequence procedure, confirmatory testing proceeds to the next hypothesis only if the preceding hypothesis meets its criterion.
This can reflect a meaningful scientific hierarchy while controlling the intended error rate.
However, the order must be established before results are known.
Researchers should not rearrange outcomes after discovering which tests are significant.
A hierarchy is most defensible when the sequence reflects substantive priorities rather than a convenient mechanism for maximizing favorable findings.
Can Correlation Between Outcomes Be Used?
Yes. Outcomes measured on the same participants are often correlated.
For example, achievement and academic self-efficacy may be positively associated.
Some multiple-testing procedures can incorporate the correlation structure and provide more efficient inference than methods that ignore it.
However, correlation does not eliminate the existence of distinct hypotheses.
Researchers should not assume that highly correlated outcomes require no multiplicity planning.
The dependence structure must be handled using an appropriate statistical method and scientifically justified hypothesis family.
What About Multivariate Statistical Tests?
Multivariate procedures may test a joint hypothesis concerning several outcomes.
For example, a multivariate analysis of variance may evaluate whether treatment groups differ on a vector of outcome means.
Such an analysis can be appropriate when the scientific objective concerns a joint multivariate effect.
However, a significant overall test does not automatically establish which individual outcomes differ.
Subsequent outcome-specific tests may require additional inferential planning.
Researchers should choose multivariate methods because they match the research question, not simply to avoid discussing multiplicity.
Does a Composite Outcome Solve the Problem?
A composite outcome may reduce the number of separate primary tests by combining several components into one endpoint.
However, the composite must represent a meaningful scientific construct or event definition.
Combining unrelated outcomes merely to obtain one p-value can obscure important differences among components.
For example, an intervention may improve a common but relatively minor component while having little effect on a more consequential component.
The interpretation of composite outcomes therefore requires conceptual and statistical justification.
Do Secondary Outcomes Require Multiplicity Adjustment?
Not every secondary outcome requires the same treatment.
Some secondary hypotheses may be included in a prespecified confirmatory hierarchy and support formal claims when the relevant testing conditions are met.
Others may be analyzed supportively or exploratorily.
For exploratory outcomes, nominal p-values should not be interpreted as though they establish independently controlled confirmatory evidence.
Researchers should distinguish the purpose of the analysis from the mere availability of a statistical test.
The decision about powering secondary outcomes should be coordinated with their inferential status.
How Does Multiplicity Planning Affect Sample Size?
Multiplicity procedures can make individual rejection thresholds more stringent, potentially increasing the sample size needed to maintain a specified level of power.
For example, detecting a treatment effect at a two-sided threshold of 0.0125 may require more information than detecting the same effect at 0.05, all else equal.
However, the relationship depends on the testing method, outcome correlations, effect sizes, and definition of overall study success.
Co-primary designs requiring success on all outcomes may need additional information to achieve acceptable joint power even when no simple alpha division is necessary.
Researchers should therefore calculate power for the actual decision procedure rather than rely on separate unadjusted calculations.
What Is the Difference Between Individual Power and Joint Power?
Individual power is the probability of rejecting a specified null hypothesis under a particular alternative.
Joint power concerns the probability of satisfying a combined success criterion.
For example, if two independent co-primary tests each have 80% power and both must succeed, the probability of overall success is:
The example illustrates why adequate power for each individual outcome does not necessarily imply adequate power for the study's overall objective.
Does Preregistration Eliminate the Need for Multiplicity Adjustment?
No. Preregistration establishes which hypotheses and analyses were planned before relevant results became available.
It improves transparency but does not independently control false-positive risk.
A study can prespecify several confirmatory outcomes and still require an appropriate multiple-testing procedure.
Similarly, a statistically valid adjustment does not compensate for poor measurement or an incoherent outcome set.
Prespecification and multiplicity control address related but distinct aspects of methodological credibility.
Can Multiplicity Be Addressed After Data Collection?
Some statistical adjustments can be calculated retrospectively, but post hoc selection of the hypothesis family or procedure creates additional interpretive concerns.
For example, choosing the adjustment method after seeing which p-values are smallest can undermine the intended confirmatory framework.
Researchers should disclose unplanned analyses and avoid presenting them as prospectively controlled confirmatory tests.
Where a study lacks adequate prespecification, exploratory interpretation and independent confirmation may be more appropriate than an unsupported definitive claim.
What About Multiple Outcomes in Educational Technology Research?
Educational technology studies frequently measure achievement, engagement, motivation, satisfaction, and self-regulation.
These outcomes may reflect different theoretical dimensions of an intervention.
However, their conceptual relevance does not automatically establish that each should support an independent confirmatory claim.
Researchers should identify the principal intervention objective and determine whether additional outcomes are necessary to establish success or primarily provide explanatory and supportive evidence.
For example, an AI tutoring system may primarily aim to improve learning achievement while engagement and satisfaction help explain implementation and user experience.
Alternatively, an intervention explicitly designed to improve both achievement and self-regulation may justify a co-primary framework.
The multiplicity strategy should follow the scientific question rather than the number of questionnaires administered.
How Should Multiplicity Be Reported?
The final report should identify the confirmatory hypothesis family, testing procedure, significance criterion, and order of any hierarchical tests.
Researchers should present the relevant effect estimates and uncertainty intervals, not merely adjusted p-values.
They should also report secondary and exploratory results transparently, including nonsignificant findings.
Changes from the prespecified analysis plan should be explained.
The objective is to allow readers to distinguish findings supported by the planned confirmatory framework from those requiring additional investigation.
Watch Out
Multiplicity planning is not simply a matter of dividing 0.05 by the number of outcomes measured. Researchers must first establish which hypotheses can support confirmatory claims, how success is defined, and which statistical error criterion is appropriate. Different outcome structures require different solutions.
