Should Sample Size Be Based Only on the Primary Outcome?

If Your Study Measures Several Outcomes, Which One Should Determine Sample Size?

You are designing an experiment to evaluate the effectiveness of AI-generated feedback in university writing courses. Your study will measure students' writing performance, feedback literacy, academic confidence, perceived usefulness, and satisfaction.

Your primary research interest is whether AI-generated feedback improves writing performance. However, you also want to examine whether the intervention changes students' confidence and perceptions of feedback quality.

When you begin calculating sample size, you discover that different outcomes require different assumptions. Writing performance is measured on a continuous scale, satisfaction is measured through questionnaire responses, and some secondary outcomes involve binary indicators.

Should you calculate sample size separately for every outcome? Should you choose whichever calculation produces the largest number? Or is it acceptable to calculate sample size using writing performance alone?

The answer depends on the inferential role assigned to each outcome. Research studies often measure several variables, but not every variable has the same importance for determining whether the study achieves its central objective.

The methodological challenge is to align sample size planning with the conclusions the study is intended to support, while distinguishing primary confirmatory claims from secondary and exploratory analyses.

The Primary Outcome Usually Drives Sample Size, but Important Exceptions Exist

How Primary, Secondary, and Co-Primary Outcomes Influence Sample Size Planning

What Is a Primary Outcome?

The primary outcome is the outcome designated as the principal basis for addressing a study's main research objective.

For example, an experiment evaluating AI-generated writing feedback may designate post-intervention writing performance as its primary outcome.

The primary outcome should be defined sufficiently precisely to support analysis and interpretation.

This includes identifying the construct, measurement instrument, assessment time point, and statistical quantity of interest.

A statement such as "student learning" may be too broad for sample size planning if it could refer to several different assessments or dimensions.

By contrast, the mean difference in scores on a specified writing assessment at the end of the intervention provides a clearer basis for statistical design.

Why Is the Primary Outcome Usually Used for Sample Size Calculation?

Sample size planning should be connected to the central scientific claim the study intends to evaluate.

When a study has one primary confirmatory hypothesis, calculating the sample required for that hypothesis provides a coherent basis for recruitment.

For example, researchers may wish to detect a five-point improvement in writing performance with 80% power using a two-sided significance level of 0.05.

The required sample then follows from the meaningful difference, anticipated variability, statistical model, and other design assumptions.

Secondary outcomes may be collected to provide additional context without independently determining the recruitment target.

This approach can prevent the sample size justification from becoming an arbitrary collection of calculations for every variable measured.

What Is the Difference Between Primary and Secondary Outcomes?

Primary and secondary outcomes differ in their intended inferential roles, not necessarily in their intrinsic scientific importance.

For example, writing performance may be primary, while feedback satisfaction and academic confidence are secondary.

Secondary outcomes can still produce valuable evidence, but their estimates may have different levels of precision and statistical power.

Researchers should not assume that an outcome becomes confirmatory merely because it was measured using a validated instrument.

Does Every Secondary Outcome Need Its Own Sample Size Calculation?

Not necessarily. A study with one primary outcome may reasonably use a single primary-outcome calculation while collecting secondary outcomes for supplementary analysis.

However, researchers should consider whether the secondary analyses are expected to support strong inferential claims.

For example, a study powered to detect a meaningful difference in writing performance may have limited power to detect a small difference in academic confidence.

That limitation should influence how the confidence findings are interpreted.

If a secondary outcome is essential to a formal decision, additional planning may be warranted even if the outcome retains a secondary label.

The designation should reflect the study's objectives rather than serve as a way to avoid necessary sample size calculations.

Does Adequate Power for the Primary Outcome Mean Adequate Power for Other Outcomes?

No. Power is specific to the hypothesis, effect size, variability, statistical test, and other assumptions.

For example, a sample of 160 students might provide adequate power for a moderately sized writing-score difference but limited power for a small difference in satisfaction ratings.

A binary secondary outcome with low prevalence may require substantially more participants than the primary continuous outcome.

Consequently, the sample size justified for one outcome cannot automatically be transferred to another.

Researchers should distinguish the study's planned confirmatory capability from the additional information that secondary analyses may provide.

See statistical power and study design for the underlying principles.

What Are Co-Primary Outcomes?

Co-primary outcomes are two or more outcomes designated as jointly central to a study's primary objectives.

For example, a university may consider an AI feedback intervention successful only if it improves writing performance and does not produce an unacceptable deterioration in feedback literacy.

The study may therefore specify more than one essential endpoint.

However, the label co-primary does not by itself establish the statistical decision rule.

Researchers must determine whether success requires favorable evidence on every co-primary outcome, whether success on any one outcome is sufficient, or whether a different prespecified rule applies.

These alternatives have different implications for statistical power and Type I error control.

How Does the Decision Rule for Co-Primary Outcomes Affect Sample Size?

Suppose a study has two co-primary outcomes.

If success requires both outcomes to meet their respective criteria, the probability of overall success depends on the joint distribution of the tests.

Powering each outcome separately at 80% does not guarantee 80% probability of meeting both criteria.

If the tests were independent and each had 80% power, the probability of both succeeding would be 0.80 × 0.80 = 0.64, or 64%.

In practice, the outcomes may be correlated, changing that probability.

If success on either outcome is sufficient, multiplicity considerations may be necessary to control false-positive conclusions across the relevant hypotheses.

Sample size planning should therefore reflect the joint success criterion rather than simply select the largest of several isolated calculations.

Should You Always Choose the Largest Sample Size from Several Outcomes?

Not automatically. Taking the largest individually calculated requirement may be reasonable in some studies, but it is not a universal solution.

For example, if several outcomes must each achieve a specified level of marginal power, the largest individual requirement may satisfy those separate targets under compatible assumptions.

However, it may not provide the desired joint power when all outcomes must be significant.

It may also be insufficient when multiple-testing adjustments change the significance thresholds.

Conversely, selecting the largest requirement for every exploratory questionnaire variable may produce an unnecessarily large and poorly justified study.

The appropriate calculation depends on the prespecified inferential decision.

How Does the Primary Outcome Affect the Statistical Test?

The measurement scale and statistical properties of the primary outcome influence the analysis and sample size calculation.

A continuous writing score may be analyzed using a mean comparison or regression model.

A binary outcome, such as achieving a competency threshold, may require a comparison of proportions or a suitable generalized linear model.

A time-to-event outcome may require assumptions about event occurrence and follow-up.

Researchers should identify the primary outcome and analysis together rather than choose a formula based solely on the word experimental.

For binary outcomes, outcome prevalence can substantially influence sample size.

What If the Primary Outcome Is Measured at Several Time Points?

A study may collect the same outcome repeatedly without necessarily having multiple primary outcomes.

For example, writing performance may be assessed at baseline, immediately after the intervention, and three months later.

The primary estimand might be the between-group difference at the final follow-up, the difference in change from baseline, or a treatment-by-time interaction.

These choices have different sample size requirements.

Researchers should specify the primary time point or longitudinal contrast before calculating sample size.

Collecting the same outcome at three occasions does not automatically mean that three independent primary tests must be powered.

See how repeated measures affect sample size requirements.

What If the Primary Outcome Is a Composite Score?

A composite outcome combines information from multiple measures or events into one defined endpoint.

For example, a study might construct a writing proficiency score from organization, argumentation, language accuracy, and evidence use.

If the composite is prespecified and appropriately validated, the sample size calculation may be based on its anticipated effect and variability.

However, researchers should not create or modify a composite after examining which components produce favorable results.

The components should also be substantively coherent.

A composite may improve efficiency in some circumstances, but it can obscure important differences among components if interpreted without care.

What If the Primary Outcome Is Difficult to Measure Reliably?

Measurement reliability can influence statistical efficiency and the credibility of the estimated effect.

For example, an essay scoring rubric applied inconsistently across raters may introduce additional measurement variation.

This can reduce power for detecting a fixed difference in writing performance.

Researchers should improve measurement procedures where feasible rather than compensate for avoidable measurement error solely through larger recruitment.

However, changing the primary outcome simply because another measure produces a smaller sample size may compromise the study's substantive purpose.

The primary outcome should remain aligned with the construct the research intends to investigate.

Can a Secondary Outcome Become Primary After Data Collection?

Changing the primary outcome after examining the results can undermine the credibility of confirmatory inference.

For example, if writing performance shows little difference but satisfaction is statistically significant, researchers should not retrospectively present satisfaction as though it had always been the primary outcome.

Such a change may introduce selective outcome reporting and distort the interpretation of statistical evidence.

Legitimate protocol amendments may occur before relevant outcome data are examined, provided they are justified and transparently documented.

Researchers should distinguish prospective changes from outcome-driven decisions.

Study registration and prespecified analysis plans can help establish the intended outcome hierarchy.

Does the Primary Outcome Need to Be the Most Statistically Sensitive Measure?

No. Statistical sensitivity is relevant, but the primary outcome should principally reflect the scientific objective.

For example, student satisfaction may be easier to measure precisely than writing performance.

However, if the intervention is intended to improve writing proficiency, satisfaction alone may not provide sufficient evidence of educational effectiveness.

Selecting an outcome because it produces a smaller required sample can create a mismatch between the research question and the eventual conclusion.

Researchers should consider substantive relevance, measurement validity, feasibility, and statistical properties together.

How Do Multiple Primary Hypotheses Affect Significance Level?

When several hypotheses can independently support a confirmatory claim, the study may need to address multiplicity.

For example, researchers may intend to claim intervention effectiveness if either writing performance or academic confidence improves.

Testing both at an unadjusted alpha of 0.05 may increase the probability of at least one false-positive conclusion under relevant null configurations.

Procedures such as Holm's method, Bonferroni adjustment, or other prespecified strategies may be appropriate depending on the hypothesis structure.

However, when success requires every co-primary endpoint to meet its criterion, the multiplicity problem differs from an any-success decision rule.

Researchers should identify the decision rule before selecting an error-control strategy.

See how significance level affects sample size.

How Should Subgroup Analyses Be Considered?

Subgroup analyses examine whether outcomes or treatment effects differ across participant categories.

For example, a researcher may investigate whether AI-generated feedback has different effects among first-year and senior students.

A sample size calculated for the overall primary outcome does not automatically provide adequate power for subgroup-specific effects or interactions.

Subgroup analyses often involve smaller numbers of participants and greater uncertainty.

If a subgroup comparison is central to the study's objectives, it should be incorporated into prospective sample size planning.

Otherwise, subgroup findings may need to be interpreted as exploratory.

What If the Study Has Several Research Questions but Only One Primary Outcome?

A study can address several research questions while retaining one primary outcome.

For example, the primary question may concern writing performance, while additional questions examine satisfaction, feedback engagement, and implementation feasibility.

These secondary questions can provide valuable context without requiring the study to establish confirmatory power for every analysis.

However, the manuscript should clearly distinguish the main objective from supplementary investigations.

Researchers should avoid presenting all research questions as equally confirmatory when only one was used to determine the sample size.

Does Every Quantitative Study Need a Primary Outcome?

Not every quantitative investigation uses a single primary endpoint in the same way.

Descriptive surveys may focus on estimating several population characteristics.

Exploratory studies may examine patterns without a single confirmatory hypothesis.

Prediction studies may prioritize model performance rather than a treatment-effect endpoint.

In such cases, sample size justification should follow the relevant estimation, modeling, or feasibility objective.

Researchers should not force a primary-outcome structure onto a study merely because it is common in randomized trials.

See whether every quantitative study needs a formal sample size calculation.

What If Different Outcomes Require Very Different Sample Sizes?

Large differences among outcome-specific requirements should prompt researchers to reconsider the study's inferential priorities.

For example, a continuous primary outcome may require 150 students, while a rare binary outcome may require substantially more observations for adequate power.

If the rare outcome is exploratory, the smaller primary-outcome-based sample may remain defensible, provided its limitations are acknowledged.

If the rare outcome is essential to the study's success, the design may need to be revised.

Researchers should not automatically ignore the larger requirement or adopt it without examining why the outcomes matter.

See what to do when different outcomes require very different sample sizes.

How Should Primary-Outcome-Based Sample Size Be Reported?

The methodology should identify the primary outcome, primary estimand, statistical test or model, and assumptions used for sample size planning.

These may include the meaningful effect size, anticipated variability or event probability, significance level, desired power, allocation, and design adjustments.

Researchers should state whether secondary outcomes were included in formal planning.

For co-primary outcomes, the joint success criterion and relevant power or multiplicity assumptions should be described.

The results should distinguish confirmatory findings from secondary or exploratory analyses.

Transparent reporting allows readers to understand which conclusions the study was prospectively designed to support.

Choosing a Sample Size When an AI Feedback Study Measures Several Outcomes

Using the conventional normal approximation for two independent means with equal allocation:

The researcher then examines the inferential role of the other outcomes.

The study can reasonably use writing performance as the basis for recruitment because it represents the principal scientific objective.

However, the researchers should not claim that the resulting sample is adequately powered for every secondary outcome.

If feedback literacy later becomes an essential co-primary endpoint, the planning problem changes. The research team would need to specify the joint decision rule and reassess the required sample.

Similarly, if 10% of recruited participants are expected to lack usable primary outcomes, the simple initial recruitment target becomes 63 / 0.90 = 70 per group, or 140 participants.

The example demonstrates how a clearly defined outcome hierarchy connects the research objective, sample size calculation, and interpretation of findings.

Common Misunderstandings About Primary Outcomes and Sample Size

How to Decide Which Outcomes Should Determine Recruitment

Begin by identifying the conclusion that the study must be able to support and the outcome most directly connected to that conclusion.

Document the outcome hierarchy before examining results. The primary analysis, sample size calculation, and main conclusion should remain aligned.

When secondary analyses are not independently powered, explain that limitation without dismissing their potential scientific value.

Before Calculating Sample Size, Establish the Outcome Hierarchy

Frequently Asked Questions About Primary Outcomes and Sample Size

Power the Study for the Conclusions It Is Designed to Support

How to Cite This Guide

This guide is intended to be read, shared, and used in research, teaching, and academic work. If you draw on its ideas, explanations, or other content, please acknowledge the source by citing the guide. Doing so gives appropriate credit and helps your readers locate the original resource.