03 · What You Need to Know
How to Determine Which Statistical Comparisons a Study Can Defensibly Support
What Counts as a Statistical Comparison?
A statistical comparison evaluates a specified relationship, difference, or contrast using observed data.
Examples include comparing mean outcomes between treatment groups, testing differences in event rates, examining treatment effects across subgroups, or evaluating several prespecified regression coefficients.
However, not every reported statistic constitutes a separate hypothesis test.
A descriptive mean, confidence interval, or graphical summary may provide useful information without representing an additional confirmatory comparison.
Researchers should distinguish estimation and descriptive analysis from formal hypothesis testing.
The relevant inferential question is which statistical procedures create opportunities to support the claims the study intends to make.
Why Is There No Universal Numerical Limit?
Studies differ in sample size, design, research objectives, and the relationships among their hypotheses.
A large multi-arm trial may support several carefully planned treatment comparisons. A small exploratory study may calculate numerous associations but provide limited precision for any individual estimate.
The number of comparisons alone does not establish whether the analysis is appropriate.
For example, ten prespecified comparisons evaluated under a valid multiplicity procedure may be more defensible than three comparisons selected after examining which results were significant.
The critical considerations are the scientific justification, statistical information, and inferential framework supporting each comparison.
What Is Multiplicity?
Multiplicity arises when a study provides several opportunities to make statistical claims, particularly claims of rejecting null hypotheses.
Examples include testing multiple outcomes, comparing several treatment arms, examining numerous subgroups, and repeatedly testing accumulating data.
When multiple true null hypotheses are tested, the probability of obtaining at least one false-positive result may exceed the nominal significance level used for an individual test.
The consequences depend on the dependence among tests and the error criterion being controlled.
Multiplicity does not mean that every analysis is invalid or that all studies must use the same adjustment.
Researchers must define which hypotheses belong to the relevant inferential family and what error properties their conclusions require.
What Is the Familywise Error Rate?
The familywise error rate, or FWER, is the probability of rejecting at least one true null hypothesis within a defined family of tests.
For confirmatory research, controlling FWER may be appropriate when several hypotheses can independently support claims of efficacy or another consequential conclusion.
For example, a study might test three experimental treatments against one control and intend to declare any treatment effective if its corresponding hypothesis is rejected.
The statistical plan must consider the possibility of false-positive claims across the relevant family.
Strong FWER control requires maintaining the intended error limit under configurations in which any subset of the null hypotheses may be true.
Weak control, by contrast, addresses a more restricted configuration, commonly the global null in which all hypotheses in the family are true.
How Does the Number of Tests Affect False-Positive Risk?
For independent tests with true null hypotheses, the probability of obtaining at least one false-positive result increases as the number of tests grows.
The calculation does not mean that ten comparisons in every research study produce a 40.1% error rate.
Tests may be correlated, may have different significance thresholds, or may be evaluated under an appropriate multiplicity procedure.
The formula illustrates why the complete testing strategy matters.
What Is the Difference Between Familywise Error Rate and False Discovery Rate?
Familywise error rate and false discovery rate address different statistical objectives.
| Criterion |
Meaning |
Typical application |
| Familywise error rate |
Probability of at least one false rejection among the tested hypotheses. |
Confirmatory settings where any false-positive claim is consequential. |
| False discovery rate |
Expected proportion of false rejections among all rejections, with the proportion defined as zero when none are rejected. |
Some high-dimensional or discovery-oriented analyses. |
| Per-comparison error rate |
False-positive probability associated with an individual test. |
Individual hypothesis evaluation, but not necessarily sufficient for a family of confirmatory claims. |
False discovery rate control can be useful when researchers investigate many potential associations and accept a specified expected proportion of false discoveries among rejected hypotheses.
However, it is not interchangeable with familywise error control.
The appropriate criterion depends on the scientific consequences of false-positive findings and the intended use of the results.
What Is the Bonferroni Correction?
The Bonferroni procedure controls familywise error by comparing each p-value with an adjusted threshold based on the number of tests.
For a family of k hypotheses and intended familywise significance level α, a simple Bonferroni threshold is α divided by k.
Bonferroni is straightforward and broadly applicable, but it can be conservative, particularly when tests are strongly correlated.
Researchers should not assume that it is always the most efficient or scientifically appropriate multiplicity procedure.
What Is the Holm Procedure?
The Holm procedure is a step-down method for controlling familywise error.
It orders the p-values from smallest to largest and compares them with progressively less stringent thresholds.
For k hypotheses, the smallest p-value is compared with α/k, the next with α/(k−1), and so forth.
Testing stops at the first comparison that fails its applicable threshold, and that hypothesis and all remaining hypotheses are not rejected.
Under valid p-values, the Holm procedure controls FWER under arbitrary dependence and is generally at least as powerful as simple Bonferroni adjustment.
However, the choice of multiplicity procedure should reflect the hypothesis structure and scientific objectives rather than convenience alone.
When Are Hierarchical Testing Procedures Appropriate?
Hierarchical or gatekeeping procedures test hypotheses in a prespecified order or structure.
For example, a trial may first test its primary outcome. Only if that test meets the specified criterion may the analysis proceed to a key secondary outcome.
Such procedures can control multiplicity while reflecting a scientifically meaningful priority among hypotheses.
However, the hierarchy must be established before relevant results are known.
Researchers should not rearrange the testing order after observing which outcomes are significant.
When the primary hypothesis fails, a fixed-sequence procedure may prevent confirmatory claims about later hypotheses, even if their nominal p-values are small.
How Do Multiple Treatment Arms Increase the Number of Comparisons?
A study with several treatment groups can generate numerous possible pairwise comparisons.
For k treatment groups, the number of distinct unordered pairwise comparisons is:
However, a four-arm study does not necessarily need to test all six comparisons.
If three experimental interventions are compared only with a shared control, the principal comparison set may contain three treatment-control hypotheses.
The scientific objectives should determine which contrasts are necessary.
Researchers should consider multiplicity planning for multiple treatment arms before finalizing the analysis.
Does Every Possible Pairwise Comparison Need to Be Tested?
No. Testing every possible contrast is often unnecessary.
For example, a study comparing three educational interventions with conventional instruction may be primarily interested in whether each intervention improves outcomes relative to the control.
Direct comparisons among the experimental interventions may not be central to the research question.
Conducting all possible comparisons can increase analytical burden and multiplicity without providing proportionate scientific value.
Researchers should specify contrasts that correspond to substantive hypotheses rather than automatically generate every comparison available in statistical software.
How Does Sample Size Affect the Number of Comparisons?
Sample size influences the precision and statistical power of individual comparisons.
A study powered for one treatment-control contrast may not be adequately powered for several smaller effects or subgroup interactions.
When multiplicity adjustments are required, the critical thresholds may become more stringent, potentially increasing the sample size needed to maintain desired power.
However, the required increase depends on the statistical procedure, effect sizes, variance, allocation, and relationships among tests.
There is no general rule that each additional comparison requires a fixed number of additional participants.
Researchers should conduct power calculations for the actual family of confirmatory hypotheses and its intended decision criteria.
Can a Study Have Enough Power for Each Test but Not for the Overall Objective?
Yes. Individual-test power and the probability of achieving a joint research objective are different quantities.
Suppose a study requires statistically significant results on two primary outcomes to declare overall success.
Even if each test has 80% power, the probability that both tests succeed may be lower than 80%, depending on their dependence.
If the tests were independent and each had 80% power, the probability of both succeeding would be:
Conversely, a design requiring success on at least one of several outcomes has a different power and multiplicity structure.
Researchers must define what constitutes study success before selecting sample size and testing procedures.
Are Planned Contrasts Better Than All Pairwise Comparisons?
Planned contrasts can improve scientific focus by aligning statistical tests with hypotheses established before data examination.
For example, researchers may hypothesize that an AI-supported intervention outperforms conventional instruction, without having a substantive reason to compare every experimental condition with every other condition.
A planned contrast can directly evaluate that hypothesis.
However, prespecification does not automatically eliminate multiplicity concerns when several planned contrasts support separate confirmatory claims.
Researchers should define the relevant hypothesis family and apply appropriate inferential procedures.
The benefit of planned contrasts lies primarily in scientific clarity and avoidance of unnecessary data-driven testing.
What About Multiple Subgroup Comparisons?
Subgroup analyses can substantially increase the number of statistical tests.
For example, researchers may examine treatment effects separately by age, sex, prior achievement, institution, and technology experience.
Each additional subgroup analysis introduces questions about statistical power, multiplicity, and whether the subgroup distinction has a defensible scientific rationale.
Importantly, a statistically significant result in one subgroup and a nonsignificant result in another does not establish that treatment effects differ between those subgroups.
Appropriate interaction testing is generally needed to evaluate effect modification.
The scientific justification for subgroup comparisons should therefore be established before extensive subgroup testing.
Do Multiple Measurement Occasions Create Additional Comparisons?
They can. A longitudinal study may measure outcomes at baseline, immediately after intervention, and at several follow-up points.
Researchers could conduct separate treatment comparisons at every occasion, but doing so may create unnecessary multiplicity and fail to use the repeated-measures structure appropriately.
A longitudinal model may provide a more coherent analysis of change over time, depending on the research question and model assumptions.
However, fitting one longitudinal model does not automatically eliminate multiplicity if researchers subsequently test numerous time-specific contrasts.
The analysis plan should identify the principal time point or treatment trajectory of interest and distinguish confirmatory from exploratory contrasts.
Are Exploratory Comparisons Subject to the Same Requirements?
Exploratory analyses have different inferential purposes from confirmatory hypothesis testing.
Researchers may examine numerous associations to identify patterns, generate hypotheses, or guide subsequent research.
Such analyses can be scientifically useful even when they are not individually powered for definitive conclusions.
However, exploratory status does not make false-positive findings irrelevant.
Researchers should report the number and nature of analyses, effect estimates, uncertainty, and any multiplicity procedures used.
Findings should be interpreted as exploratory rather than retrospectively promoted to confirmatory status because they are statistically significant.
Does Preregistration Eliminate Multiplicity Problems?
No. Preregistration helps establish which hypotheses and analyses were planned before relevant results became available.
However, prespecifying twenty confirmatory tests does not automatically control the probability of false-positive claims across them.
The statistical plan must still address the appropriate error criterion and testing structure.
Likewise, registering an unnecessarily large set of comparisons does not establish their scientific relevance.
Preregistration improves transparency, but it does not replace hypothesis prioritization, power planning, or valid multiplicity procedures.
Can Researchers Use One Overall Test Instead of Many Pairwise Tests?
Sometimes. An omnibus test may evaluate whether there is evidence of any difference among several groups or parameters.
For example, an analysis of variance may test whether all group means are equal.
However, rejecting the omnibus null hypothesis does not identify which specific groups differ.
Subsequent contrasts may be needed to answer treatment-specific questions.
Researchers should therefore distinguish an overall hypothesis from the specific comparisons required for interpretation.
An omnibus test can be useful, but it does not automatically resolve every multiplicity issue associated with subsequent analyses.
Can Correlation Between Tests Reduce the Multiplicity Burden?
Yes, dependence among tests can affect the distribution of their joint evidence and the efficiency of some multiplicity procedures.
For example, several highly correlated outcomes may produce less independent information than the same number of unrelated outcomes.
Some procedures can use the correlation structure to improve power while controlling the intended error rate.
However, researchers should not simply assume that correlated tests require no multiplicity planning.
The statistical procedure must appropriately account for the dependence structure and inferential objectives.
What Is the Risk of Data Dredging?
Data dredging occurs when researchers search extensively across variables, models, subgroups, or comparisons and selectively emphasize favorable findings without adequately accounting for the search process.
For example, a researcher might examine dozens of combinations of outcomes and participant characteristics, then report only the comparisons with p-values below 0.05.
This can create a misleading impression that the significant findings were anticipated and independently supported.
The problem is not exploration itself. It is presenting selected exploratory results as though they arose from a limited, prespecified confirmatory testing procedure.
Transparent reporting and clear distinctions between confirmatory and exploratory analyses are essential.
When Should a Study Be Divided Into Separate Research Questions?
A study may become analytically incoherent when its comparisons represent numerous unrelated scientific objectives.
For example, one project might simultaneously investigate intervention effectiveness, technology acceptance, institutional policy, and long-term employment outcomes.
These questions may require different designs, populations, measurement schedules, and analytical frameworks.
Dividing them into distinct research components may improve scientific clarity.
However, separating analyses into multiple publications does not automatically remove multiplicity concerns when the underlying testing process and inferential claims remain connected.
Researchers should preserve transparency about shared datasets, overlapping hypotheses, and related analyses.
How Should Researchers Determine the Right Number of Comparisons?
Begin with the substantive hypotheses rather than the statistical tests available in software.
Identify the comparisons necessary to answer the principal research question and distinguish them from supportive or exploratory analyses.
Then define the relevant hypothesis families, intended error criteria, and power requirements.
Finally, assess whether the sample size and measurement quality support the intended conclusions.
The appropriate number of comparisons is the number that can be scientifically justified and statistically interpreted within a coherent analytical framework.
Watch Out
Conducting many statistical tests is not inherently inappropriate, but reporting only significant results or treating every nominal p-value below 0.05 as an independent confirmatory discovery can substantially misrepresent the evidence. The full comparison strategy and intended inferential claims must be considered.
