03 · What You Need to Know
How to Reconcile Conflicting Sample Size Calculations Without Weakening the Study
Why Can Different Outcomes Require Very Different Samples?
Sample size requirements depend on the statistical properties of the outcome and the inference researchers intend to make.
For example, a continuous writing score may provide information about differences across a broad range of performance levels.
A binary outcome indicating whether a student achieved a competency threshold contains different information, particularly when one category is uncommon.
Similarly, a small meaningful difference in academic confidence may be more difficult to detect than a larger standardized difference in writing performance.
Different outcomes may therefore require different sample sizes even when they are measured from the same participants.
The disagreement among calculations may reflect genuine differences in effect magnitude, variability, event frequency, statistical model, or desired precision.
What Should You Check Before Comparing the Calculations?
Before deciding which requirement should determine recruitment, verify that the calculations address clearly defined objectives and use consistent design assumptions.
For example, one calculation may use 80% power while another uses 90% power.
One may assume a two-sided alpha of 0.05, while another applies a stricter threshold for multiple comparisons.
Some calculations may account for clustering or attrition, while others may not.
These differences can make the numerical requirements difficult to compare directly.
Researchers should identify which assumptions differ because of the outcome itself and which differ because the calculations were prepared inconsistently.
Should You Always Use the Largest Calculated Sample Size?
No. Using the largest individual requirement may be appropriate when every corresponding outcome must meet a specified marginal power or precision target.
However, this strategy can be unnecessarily demanding when the largest requirement belongs to an exploratory outcome.
For example, a rare binary outcome may require 800 participants, while the study's primary scientific objective concerns a writing-performance difference that requires 150.
If the rare outcome is supplementary, recruiting 800 participants may not be justified solely by its inclusion.
Conversely, the largest individual calculation may be insufficient when the study requires simultaneous success across several outcomes.
The decision depends on the intended inferential claims, not merely on the numerical maximum.
Should You Use the Smallest Sample Size to Keep the Study Feasible?
Not unless the smallest requirement corresponds to the study's legitimate primary objective and other essential analyses are adequately addressed.
For example, choosing 150 participants because that is the smallest calculated requirement would be inappropriate if the study's main purpose is to establish an effect on a rare competency outcome requiring substantially more information.
Recruitment feasibility is important, but it does not establish statistical adequacy.
Researchers should not redefine an essential outcome as exploratory solely to justify an affordable sample after seeing an inconvenient calculation.
Changes to the outcome hierarchy should follow scientific reasoning and be documented before relevant results are examined.
How Does the Primary Outcome Help Resolve the Conflict?
When one outcome clearly represents the principal confirmatory objective, its sample size requirement ordinarily provides the starting point.
For example, if the study is primarily designed to evaluate improvement in writing performance, the writing-performance calculation may determine recruitment.
Other outcomes may then be included as secondary or exploratory measures.
However, the researchers should acknowledge that the study may have limited power or precision for some of those outcomes.
The primary outcome should be selected because of its substantive importance, not because its calculation is the most convenient.
See whether sample size should be based only on the primary outcome.
What If Two or More Outcomes Are Equally Essential?
If several outcomes are indispensable to the study's main conclusion, the researchers may need a co-primary or multiple-primary-outcome framework.
For example, a university may require evidence that AI feedback improves writing performance while preserving a specified level of academic integrity.
The study's success criterion should explain whether both conditions must be satisfied or whether either one is sufficient.
These decision rules have different statistical implications.
If every co-primary outcome must succeed, the study may require sufficient joint power.
If success on any one outcome is sufficient, the analysis may need multiplicity control.
Researchers should calculate sample size for the complete decision rule rather than rely exclusively on separate endpoint calculations.
Why Is Joint Power Different from Individual Power?
Individual power describes the probability of rejecting a specified null hypothesis under its corresponding alternative.
Joint power may describe the probability that several required tests all meet their success criteria.
Suppose a study requires two co-primary outcomes to be significant.
If each test has 80% power and the test outcomes are independent, the probability that both succeed is:
Therefore, choosing a sample size that provides 80% power for each outcome separately does not necessarily produce 80% overall power for a decision requiring both.
Joint calculations or simulation may be necessary.
How Does Multiple Testing Change the Required Sample?
When several confirmatory hypotheses are evaluated, the statistical testing strategy may affect the required sample.
For example, a study may claim success if any one of three outcomes shows a favorable effect.
Using an unadjusted alpha of 0.05 for every test can increase the probability of false-positive conclusions across the relevant hypothesis family.
Multiplicity procedures may impose more stringent rejection criteria.
Maintaining the same power under stricter thresholds generally requires additional observations.
However, multiplicity requirements depend on the hypotheses and decision rule. They should not be applied mechanically without defining the inferential family.
See how alpha affects sample size.
How Can You Compare Requirements Across Different Outcome Types?
Researchers should create a planning summary that identifies the statistical objective and assumptions for each outcome.
For continuous outcomes, this may include the meaningful mean difference and anticipated standard deviation.
For binary outcomes, it may include baseline prevalence and the meaningful difference or relative effect.
For repeated measurements, the correlation structure and primary time point or contrast may be relevant.
For clustered studies, the number of clusters and intraclass correlation should be considered.
Comparing these assumptions can reveal whether an apparently large sample requirement reflects a genuinely demanding objective or an inappropriate planning method.
Can You Change the Statistical Analysis to Reduce Sample Size?
Sometimes a more efficient statistical analysis can reduce the required sample without weakening the scientific objective.
For example, adjusting a continuous follow-up outcome for a strongly prognostic baseline measurement may improve precision in a randomized study.
Repeated-measures methods may also use within-person information efficiently when the analysis and correlation structure support that approach.
However, statistical efficiency gains depend on assumptions that should be justified using credible evidence.
Researchers should not change the analysis solely to obtain a favorable calculation without considering whether it answers the intended question.
See how repeated measures affect sample size.
Can You Choose a More Sensitive Outcome Measure?
Potentially. A reliable and responsive measurement instrument may provide more information than a noisy or poorly targeted measure.
For example, a detailed writing rubric may detect meaningful changes that a crude pass-or-fail indicator cannot distinguish.
However, the two measures may represent different estimands.
A mean writing-score difference is not equivalent to the probability of achieving a competency threshold.
Researchers should therefore evaluate whether the alternative measure preserves the scientific meaning of the original objective.
Improving measurement quality is defensible; replacing an essential outcome with an easier one solely to reduce sample size may not be.
Can You Replace a Binary Outcome with a Continuous Outcome?
Sometimes, but the decision changes the parameter being studied.
For example, writing proficiency may be measured continuously or classified according to whether a student meets a competency threshold.
Dichotomizing a continuous outcome often discards information and can reduce statistical efficiency.
However, a competency threshold may be substantively important for educational policy.
If the research question concerns the proportion of students meeting a standard, replacing that outcome with a mean score may fail to answer the original question.
Researchers should distinguish statistical efficiency from the relevance of the outcome to the decision being investigated.
Can You Reduce the Required Sample by Assuming a Larger Effect?
Mathematically, larger assumed effects generally require fewer observations to detect under common testing designs.
However, choosing an optimistic effect merely to make the study feasible is not a defensible solution.
For example, a study may require 500 students to detect a meaningful three-point writing improvement but only 125 to detect a six-point improvement under a simplified independent-means approximation.
The smaller calculation does not establish that the six-point effect is likely or that smaller meaningful effects can be ignored.
Researchers should use substantive thresholds and credible evidence to justify effect assumptions.
See how effect size affects sample size.
What If One Outcome Is Rare?
Rare binary outcomes may require substantially larger samples because relatively few events occur.
For example, a study may have sufficient participants to detect a writing-score difference but too few instances of a serious academic integrity incident for stable regression modeling.
Researchers should evaluate whether the rare outcome is essential, secondary, or exploratory.
If it is essential, the design may need a larger sample, longer observation period, or an alternative recruitment strategy.
However, deliberate oversampling of cases may change the population inference and require specialized analysis.
See how outcome prevalence affects sample size.
Can a Composite Outcome Resolve Different Sample Size Requirements?
A composite outcome may combine several related measures into one prespecified endpoint.
For example, writing quality might be represented through a validated composite of organization, argumentation, and evidence use.
Such an approach may be useful when the components collectively represent one coherent construct.
However, combining unrelated outcomes solely to reduce the sample size can obscure meaningful differences and create interpretation problems.
Researchers should justify the composite's substantive meaning, scoring, and statistical properties before data collection.
A composite is not an automatic solution to conflicting sample size calculations.
Should You Drop Outcomes That Require Too Many Participants?
Not automatically. Removing an outcome may simplify the study, but it may also weaken the research question.
For example, excluding an academic integrity outcome may be inappropriate if institutional adoption of AI feedback depends on both educational effectiveness and acceptable integrity-related consequences.
Alternatively, a supplementary satisfaction measure may be retained without a separate confirmatory power target.
The decision should consider the scientific role of each outcome and whether its measurement adds meaningful evidence.
Researchers should not remove outcomes merely because their expected results are uncertain or inconvenient.
Can You Retain Underpowered Secondary Outcomes?
Yes, provided their inferential role and limitations are reported accurately.
For example, a study powered for writing performance may still estimate changes in academic confidence.
The confidence estimates may be useful for generating hypotheses or informing future research.
However, a nonsignificant result should not be interpreted as evidence that the intervention has no effect on confidence.
Likewise, a statistically significant secondary result should be interpreted in light of multiplicity, precision, and the prespecified analysis plan.
Secondary outcomes remain valuable when their evidence is not overstated.
How Does Precision-Based Planning Change the Decision?
Not every study is designed around hypothesis testing.
For descriptive research, different outcomes may require different sample sizes to achieve desired confidence interval widths.
For example, estimating the prevalence of weekly AI use within five percentage points may require a different sample from estimating a rare behavior within one percentage point.
Researchers should identify which estimates require a particular level of precision.
If several population estimates are equally essential, the sample size may need to satisfy multiple precision targets.
However, a precision requirement should not be selected without considering the practical usefulness of the resulting interval.
What If the Largest Required Sample Is Impossible to Recruit?
An infeasible sample size should prompt a reconsideration of the study's objectives, design, and resources.
Possible responses include narrowing the confirmatory objective, improving measurement efficiency, extending recruitment, collaborating across institutions, or conducting a preliminary study.
However, each option has consequences for feasibility, inference, and scientific scope.
Researchers should not present a smaller feasible sample as adequately powered for an essential outcome when the calculation indicates otherwise.
See what to do when the calculated sample size is impossible to recruit.
Can a Pilot or Feasibility Study Be a Better Alternative?
Potentially. When the available recruitment capacity cannot support a definitive confirmatory investigation, a preliminary study may be appropriate.
For example, a pilot may estimate recruitment yield, outcome variability, assessment completion, and implementation feasibility.
These findings can inform a later adequately planned study.
However, a pilot should not be presented as a definitive effectiveness evaluation merely because it uses the same intervention and outcome measures.
The sample size justification should reflect the pilot's actual objectives.
How Can Sensitivity Analysis Help Resolve Conflicting Requirements?
Sensitivity analysis examines how sample size requirements change under plausible assumptions.
For example, the researcher may vary the meaningful effect, outcome standard deviation, baseline event probability, or expected retention.
This can reveal whether the difference between outcome-specific requirements is robust or driven by uncertain assumptions.
However, sensitivity analysis should not be used to search selectively for the assumptions that produce the smallest sample.
The plausible range should be justified through substantive knowledge and available evidence.
When several outcomes must satisfy a joint decision rule, simulation may be particularly useful for evaluating overall power.
Should You Change the Outcome Hierarchy After Seeing Sample Size Results?
Revising the design during prospective planning can be legitimate when the change is scientifically justified.
For example, a research team may recognize that its original objectives are too broad for the available resources and decide to focus on one clearly defined primary outcome.
However, the revised hierarchy should reflect a coherent scientific purpose, not merely the desire to avoid an inconvenient calculation.
The rationale should be documented before relevant outcome data are examined.
Changing priorities after observing results creates additional concerns about selective reporting and confirmatory interpretation.
How Should Different Sample Size Requirements Be Reported?
The methodology should identify the outcomes considered during planning and explain which calculations determined the final recruitment target.
For essential outcomes, report the relevant effect sizes, variability or event probabilities, significance levels, power or precision targets, and analytical assumptions.
Where several hypotheses form a joint decision rule, describe the corresponding calculation or simulation.
If some outcomes are secondary or exploratory, explain that their precision or power may differ from the primary analysis.
When feasibility constraints require a smaller sample than an essential calculation indicates, acknowledge the limitation explicitly.
Transparent reporting is preferable to presenting one convenient sample size as universally adequate for every outcome.
