03 · What You Need to Know
Why Outcome Proliferation Threatens Credible Statistical Inference
What Is Outcome Proliferation?
Outcome proliferation occurs when a study accumulates numerous outcome variables, measurements, time points, subscales, or related hypotheses without a sufficiently clear justification or analytical hierarchy.
There is no universal numerical threshold at which a study has too many outcomes. Ten well-justified outcomes may be entirely appropriate for a comprehensive investigation, whereas three poorly defined outcomes may already create substantial ambiguity.
The critical issue is the relationship between the number of outcomes and the questions the study is designed to answer. A study becomes difficult to interpret when researchers cannot explain which findings are central, which are supportive, and which are exploratory.
Outcome proliferation is particularly concerning when researchers have considerable freedom to choose what to report after observing the data.
How Does Testing Many Outcomes Increase False-Positive Opportunities?
Every statistical hypothesis test carries some probability of rejecting a true null hypothesis. When researchers perform numerous tests and interpret each at an unadjusted significance threshold, the probability of obtaining at least one false-positive result can increase.
For illustration, suppose all null hypotheses are true and 20 independent tests are conducted at a significance level of 0.05.
Under these assumptions, the probability of observing at least one false-positive result is approximately 64.2%. This does not mean that 64.2% of published findings are false, nor does it describe the probability for every study with 20 outcomes. Correlations among tests, the truth of individual hypotheses, and the testing procedure all affect the calculation.
The example demonstrates why a nominal p-value below 0.05 cannot always be interpreted independently of the broader testing strategy. Researchers should determine how many statistical comparisons their study can reasonably support before conducting confirmatory analyses.
Why Is Outcome Proliferation More Than a Multiplicity Problem?
Multiplicity concerns the statistical consequences of examining several hypotheses. Outcome proliferation also creates opportunities for selective reporting, selective interpretation, and post hoc changes to research questions.
A researcher might measure engagement at three time points, calculate several subscale scores, compare multiple participant groups, and examine both adjusted and unadjusted models. Even when the study appears to have only one broad outcome domain, these choices may generate many candidate results.
The Cochrane Handbook distinguishes the outcome domain, its specific measurement, and the statistical analysis used to estimate an effect. This distinction matters because selective reporting can occur at any of these levels.
For example, researchers might report the most favorable engagement subscale, choose the time point with the smallest p-value, or emphasize one analytical model while omitting alternatives. The resulting report may appear more conclusive than the full evidence warrants.
How Do Outcome Proliferation, Selective Reporting, and p-Hacking Differ?
| Concept |
Central problem |
Illustrative example |
| Outcome proliferation |
Too many insufficiently prioritized outcomes or outcome-related analyses |
Measuring numerous overlapping indicators without explaining their scientific roles |
| Selective outcome reporting |
Reporting or emphasizing outcomes according to their observed results |
Omitting prespecified outcomes that did not show favorable effects |
| Outcome switching |
Changing outcome definitions or priorities, potentially after examining results |
Presenting an originally secondary outcome as primary without transparent disclosure |
| p-Hacking |
Using analytical flexibility to obtain or preferentially present statistically significant results |
Trying alternative exclusions, transformations, or models and reporting only the favorable analysis |
| HARKing |
Presenting hypotheses developed after results are known as though they were specified beforehand |
Claiming that an unexpected subgroup effect was the original hypothesis |
These practices can overlap, but they are not identical. Measuring many outcomes is not itself evidence of questionable research conduct. Problems arise when analytical flexibility and selective presentation undermine the distinction between what was planned and what was discovered.
How Should Researchers Decide Which Outcomes Are Necessary?
Begin with the substantive research question rather than the list of variables that could conveniently be collected. Every proposed outcome should have an identifiable role in answering that question.
A useful classification distinguishes primary outcomes, key secondary outcomes, other secondary outcomes, and exploratory outcomes. These categories should reflect the intended scientific claims rather than merely the order in which variables appear in the questionnaire.
The designation of a primary outcome should clarify the principal objective and, where applicable, the primary hypothesis and sample size justification.
Secondary outcomes may provide complementary evidence about mechanisms, benefits, harms, or other relevant consequences. Exploratory outcomes can investigate emerging questions without requiring every result to support a definitive claim.
Researchers should also examine whether several measures are genuinely distinct. Collecting multiple instruments for the same construct may be appropriate for validation or triangulation, but redundant measures should not automatically become separate opportunities for confirmatory significance.
What Must Be Prespecified Before Data Collection?
Prespecification should identify the outcomes relevant to the planned claims and describe how they will be measured and analyzed. For a confirmatory quantitative study, this generally includes the outcome variable, measurement instrument, assessment time point, participant-level summary, comparison, and statistical method.
Where relevant, the protocol should also describe the handling of missing data, covariate adjustment, subgroup analyses, and the intended approach to multiplicity.
SPIRIT 2025 provides a contemporary framework for documenting randomized trial protocols, while CONSORT 2025 addresses transparent reporting of completed randomized trials. These guidelines are specifically designed for trials, although their underlying transparency principles can inform other research designs.
Prespecification is most useful when sufficiently detailed to distinguish the planned analysis from plausible alternatives. A statement such as "student engagement will be evaluated" leaves considerable uncertainty about the instrument, time point, subscale, and statistical comparison.
Does Preregistration Prevent Selective Reporting?
Preregistration can substantially improve transparency by creating a dated record of intended hypotheses, outcomes, and analytical decisions before relevant results are known. It does not, by itself, guarantee methodological quality or prevent every form of selective reporting.
A vague preregistration may still leave many analytical choices unresolved. Researchers may also legitimately revise protocols when circumstances change, provided the timing and reasons are documented.
For randomized trials, the protocol, registry entry, statistical analysis plan, and final report should be consistent or contain transparent explanations of deviations. Researchers working in other disciplines can adopt comparable documentation practices appropriate to their designs.
Must Researchers Adjust for Every Outcome They Measure?
No. Multiplicity adjustment should be determined by the inferential objectives and the structure of the hypotheses, not simply by counting every recorded variable.
Several outcomes may belong to one confirmatory family, while other analyses are descriptive, exploratory, or concerned with safety. These distinctions can justify different statistical treatments.
When multiple outcomes are intended to support confirmatory conclusions, researchers should consider an appropriate prospective multiplicity strategy. The choice may involve familywise error control, hierarchical testing, or another procedure justified by the study's objectives.
Multiplicity adjustment does not repair selective reporting. An adjusted p-value cannot compensate for choosing the outcome definition after seeing which result is most favorable.
Can Researchers Still Conduct Exploratory Analyses?
Yes. Exploration is an essential part of scientific inquiry. Unexpected patterns can suggest mechanisms, identify overlooked populations, or motivate new research questions.
The distinction lies in how results are presented. A confirmatory analysis evaluates a hypothesis using a prespecified inferential framework. An exploratory analysis investigates possibilities whose interpretation should acknowledge analytical flexibility and uncertainty.
Exploratory findings should not be suppressed merely because they were unplanned. They should be labeled accurately, accompanied by relevant effect estimates and uncertainty, and interpreted without implying that they have undergone the same evidential test as prespecified confirmatory hypotheses.
How Can Complete Reporting Reduce Publication Bias?
Selective reporting can distort both individual study conclusions and the cumulative research literature. If favorable outcomes are published while unfavorable or inconclusive outcomes remain unavailable, later systematic reviews may inherit a misleading evidence base.
CONSORT 2025 emphasizes transparent reporting of prespecified outcomes and important changes to outcomes or analyses. The Cochrane Handbook likewise identifies selective non-reporting and selection of reported results as important sources of bias.
Researchers should report all prespecified outcomes with sufficient detail for interpretation, including estimates and measures of uncertainty where appropriate. Supplementary materials or repositories can accommodate extensive results without overwhelming the main manuscript.
Complete reporting does not require treating every outcome as equally important. It requires making the distinction between importance and availability explicit.
04 · A Practical Example
How a Study With Many Outcomes Can Produce a Misleading Positive Conclusion
Hypothetical Example
Evaluating an AI-Supported Learning Intervention
A university conducts a randomized study comparing an AI-supported learning intervention with conventional instruction. The research team measures 20 outcomes covering academic achievement, engagement, motivation, self-regulation, satisfaction, and related constructs.
The team initially treats all 20 tests as separate opportunities to demonstrate effectiveness, without specifying a primary outcome or a multiplicity strategy. After analysis, two outcomes have nominal p-values below 0.05, while the remaining 18 do not.
Identify the original analytical problem.
The study lacks a clearly defined principal hypothesis. The researchers could selectively emphasize whichever of the 20 outcomes produces the most favorable result.
Recognize the statistical uncertainty.
With numerous unadjusted tests, nominal significance is less persuasive than it would be for one prespecified test. The two significant results may reflect genuine effects, chance variation, or a combination of both.
Consider how the study should have been planned.
Before examining results, the researchers could have designated academic achievement as the primary outcome, identified theoretically important secondary outcomes, and classified remaining analyses as exploratory.
Report the existing study transparently.
Because the original design did not specify that hierarchy, the researchers must not retrospectively portray it as prespecified. They should disclose the full outcome set, explain the analytical limitations, and interpret the significant results cautiously.
A scientifically defensible conclusion might state that the study found exploratory evidence of differences in two measured outcomes, while the broader pattern did not establish a consistent intervention effect across the evaluated domains.
This conclusion does not dismiss the two findings. It places them within the complete evidence and avoids presenting an unplanned selection of favorable results as a definitive demonstration of effectiveness.
If the study were being designed prospectively, the research team could also examine whether some outcomes should be removed, treated descriptively, or combined into a validated measure. The latter option would require substantive and measurement justification, not merely a desire to reduce the number of tests.
06 · What This Means for You
How to Keep Outcome Selection Focused Without Restricting Scientific Discovery
Before collecting data, construct an outcome inventory that connects each measure to a specific research objective. Identify which outcomes are essential for answering the principal question and which provide additional or exploratory information.
If the inventory becomes extensive, reconsider whether the study is attempting to answer several independent questions. The issue may concern the scope of the research itself rather than the statistical analysis alone.
A simple decision framework
If an outcome directly addresses the principal research question
Consider designating it as primary and align the study design, sample size, and analysis with that decision.
If several outcomes support distinct confirmatory claims
Prespecify their hypothesis structure and a suitable multiplicity strategy.
If outcomes are useful but not essential to the primary claim
Retain them as justified secondary, descriptive, or exploratory outcomes with appropriate interpretation.
If an unexpected association emerges after analysis
Report it transparently as exploratory and consider independent confirmation.
If the project contains several largely independent objectives
During analysis, maintain a record of deviations from the original plan. Distinguish prespecified tests from additional analyses and document why changes were necessary.
When writing, organize the results according to the research objectives rather than statistical significance. Give the primary findings appropriate prominence, report the complete prespecified outcome set, and avoid framing unexpected favorable findings as the original basis for the study.
Researchers should be able to answer a simple question: would the study's central conclusion have been framed the same way if the most favorable secondary result had not been statistically significant? If not, the interpretation may be excessively dependent on post hoc outcome selection.
