Should You Revise the Research Question, Design, Outcome, or Effect Size Assumption When the Required Sample Size Is Unrealistic?

Which Part of Your Research Should Change When the Sample Size Is Too Large?

You are preparing an experiment to determine whether AI-generated feedback improves university students' writing performance. Your sample size calculation indicates that approximately 500 students are needed to detect the smallest educationally meaningful improvement with the desired statistical power.

However, you can recruit only 150 students.

You consider several possibilities. You could narrow the research question, introduce baseline measurements, use a different writing assessment, or increase the assumed effect size until the calculated requirement becomes manageable.

Each option could change the sample size calculation, but not necessarily for the same reason.

Some changes improve statistical efficiency while preserving the original scientific question. Others change what the study is designed to estimate or test. Still others merely make the calculation appear feasible without improving the underlying research.

The methodological challenge is to determine which revisions are scientifically defensible, what each revision sacrifices or improves, and when the original research objective should be retained despite its recruitment demands.

Revise the Element That Creates the Mismatch, Not the Assumption That Is Easiest to Change

How to Decide Whether the Research Question, Design, Outcome, or Planning Assumptions Should Change

Why Is an Unrealistic Sample Size a Research Design Problem?

Sample size is not an isolated administrative requirement. It follows from the scientific question, estimand, statistical analysis, desired precision or power, and assumptions about the data.

For example, detecting a small difference between two groups may require substantially more participants than detecting a large difference under otherwise comparable conditions.

Similarly, a study comparing rare binary outcomes may require a different sample from one estimating a continuous writing score.

When the required sample is unattainable, the researcher should examine how these elements interact.

The difficulty may arise from an incorrect calculation, an inefficient design, an overly ambitious set of objectives, or a genuine shortage of eligible participants.

Different causes warrant different responses.

What Should Be Reconsidered First?

Before changing the scientific objective, verify the original sample size calculation.

Check that the planned statistical test matches the research design and that the correct effect measure, variance, significance level, and power assumptions were entered.

For example, a researcher may have calculated sample size for independent groups even though the intended analysis adjusts for a highly prognostic baseline score.

Alternatively, a cluster-randomized study may have omitted the intraclass correlation or applied the clustering adjustment twice.

These problems should be corrected before evaluating more substantial revisions.

However, a correct calculation should not be repeatedly altered until it produces the desired number.

How Do You Distinguish Revising the Question from Revising the Design?

A research question defines what the study seeks to learn. A research design specifies how evidence will be generated to address that question.

For example, changing from a definitive effectiveness question to a feasibility question changes the objective.

Adding a reliable baseline assessment to an otherwise comparable randomized experiment may improve the design while retaining the treatment-effect objective.

Researchers should determine which type of change is necessary rather than treating all modifications as equivalent sample size adjustments.

When Should You Revise the Research Question?

Revising the research question may be appropriate when the original objective cannot be answered credibly with the available information and no feasible design improvement resolves the limitation.

For example, a researcher may initially ask whether AI-generated feedback produces a small but educationally meaningful improvement across a broad university population.

If recruitment capacity is insufficient, the study might instead investigate implementation feasibility, student engagement, or the precision of preliminary effect estimates.

However, the revised question should be independently worthwhile.

Researchers should not simply weaken the wording of a confirmatory question while retaining conclusions that imply definitive effectiveness.

A genuine revision changes the inferential objective and the claims the study will support.

Does Narrowing the Research Question Always Reduce Sample Size?

No. A narrower question may improve feasibility, but it does not necessarily reduce the number of observations required for a specified statistical effect.

For example, restricting a study to first-year students may improve recruitment coordination and reduce some contextual heterogeneity.

However, if the meaningful standardized effect and other assumptions remain unchanged, the statistical requirement may remain similar.

Narrowing the population can also reduce the number of eligible participants, making recruitment more difficult.

Researchers should therefore distinguish a more focused scientific question from a statistically more efficient design.

When Should You Revise the Research Design?

Design revision is particularly attractive when it can improve statistical efficiency while preserving the intended estimand.

For example, a randomized study comparing AI feedback with conventional feedback may use a baseline-adjusted analysis rather than a simple unadjusted post-intervention comparison.

If baseline writing proficiency strongly predicts follow-up performance, adjustment may reduce residual variability.

Other possibilities include appropriate blocking, improved allocation, or more reliable measurement procedures.

However, each modification should be evaluated using the actual planned analysis and credible assumptions.

Researchers should not assume that a more complicated design is automatically more efficient.

Can Baseline Adjustment Reduce the Required Sample?

Potentially. In randomized studies with continuous outcomes, adjusting for a prognostic baseline measurement may improve precision.

Under a simplified linear model, the residual variance of a follow-up outcome after adjustment for a baseline predictor is proportional to 1 − ρ², where ρ is the baseline-follow-up correlation.

However, the exact sample size benefit depends on the analysis, allocation, measurement reliability, and other assumptions.

The researcher should recalculate power for the proposed baseline-adjusted model rather than automatically reduce the original sample by 36%.

Should You Change from an Independent-Groups Design to a Repeated-Measures Design?

Sometimes, but the revised design must still answer the intended question.

Repeated measurements can improve precision when within-person information is relevant and appropriately modeled.

For example, collecting baseline and follow-up writing scores may help estimate change or improve an adjusted treatment comparison.

However, replacing a randomized comparison with a one-group pre-post study changes the causal evidence available.

An observed improvement may reflect maturation, practice effects, or other influences rather than the intervention.

Researchers should not sacrifice essential causal identification merely to reduce the required sample.

See how repeated measures affect sample size requirements.

When Should You Reconsider the Primary Outcome?

Outcome revision may be justified when the original measure is unreliable, poorly aligned with the research construct, or unnecessarily inefficient.

For example, a binary pass-or-fail indicator may discard meaningful information available in a validated continuous writing score.

Using the continuous score could improve statistical efficiency in some circumstances.

However, the change may also alter the scientific question.

If the study's policy objective concerns the proportion of students achieving a competency standard, a mean-score difference is not an equivalent substitute.

Researchers should preserve the substantive meaning of the outcome rather than select whichever measure produces the smallest sample size.

Can a More Reliable Measurement Instrument Reduce Sample Size?

Potentially. Measurement error may increase unexplained variation and reduce statistical efficiency.

For example, a poorly specified essay rubric may produce inconsistent scores across raters.

Improving the rubric, establishing scoring procedures, and evaluating inter-rater reliability may strengthen the outcome measure.

However, improved reliability does not guarantee a fixed reduction in required participants.

The statistical consequences depend on how measurement error affects the estimand and analysis.

Researchers should prioritize measurement validity and reliability, then estimate the resulting precision under defensible assumptions.

Can You Change the Outcome Time Point?

Changing the primary assessment time point may alter the expected effect, variability, retention, and interpretation.

For example, AI-generated feedback might produce a larger immediate improvement than a sustained improvement three months later.

Using immediate post-intervention scores may require fewer participants if the anticipated effect is larger.

However, if the research question concerns durable learning, replacing delayed follow-up with an immediate assessment changes the objective.

The primary time point should follow the educational process and intended inference.

It should not be chosen solely because the expected effect is more favorable.

When Is Revising the Expected Effect Size Justified?

Effect size assumptions may be revised when new, credible evidence changes what researchers reasonably expect or when the study's substantively meaningful target effect is reconsidered with justification.

For example, a systematic review may reveal that the original planning estimate came from a study using a substantially different intervention or population.

A revised estimate based on more comparable evidence may be appropriate.

Similarly, stakeholders may establish a defensible threshold for the smallest educationally meaningful effect.

However, an effect assumption should not be changed merely because the original value produces an infeasible sample size.

See how effect size affects sample size.

What Is the Difference Between an Expected Effect and a Minimum Meaningful Effect?

The expected effect describes what researchers anticipate may occur under the proposed conditions.

The minimum meaningful effect describes the smallest magnitude that would matter for the scientific or practical decision.

For example, researchers may expect a six-point writing improvement while considering a three-point improvement educationally worthwhile.

A study powered only for six points may have limited power to detect the smaller but still meaningful three-point effect.

The two quantities should not be conflated merely to obtain a feasible sample.

Why Is Inflating the Expected Effect Methodologically Problematic?

In many common designs, required sample size decreases as the assumed effect increases.

For a conventional two-group mean comparison, the approximate sample is inversely proportional to the square of the standardized effect.

Consequently, doubling the planning effect may reduce the required sample to approximately one quarter under otherwise fixed assumptions.

However, this mathematical reduction does not improve the study's ability to detect the original smaller effect.

It merely changes the alternative hypothesis used for planning.

If the larger effect lacks a defensible basis, the revised calculation can create an unjustified impression of statistical adequacy.

Should You Revise the Significance Level or Power Target?

Alpha and desired power are legitimate design choices, but changing them alters the statistical operating characteristics of the study.

A less stringent alpha increases the nominal probability of a Type I error under the relevant testing assumptions.

A lower power target increases the probability of failing to reject the null hypothesis when the specified alternative is true.

These tradeoffs may be considered in particular research contexts, but they should be scientifically justified and prospectively documented.

They should not be used merely to make an infeasible design appear equivalent to the original one.

See statistical power in study design and how alpha affects sample size.

What If the Study Has Several Primary or Secondary Outcomes?

Multiple outcomes may create competing sample size requirements.

For example, writing performance may require fewer participants than a rare binary academic integrity outcome.

Researchers should determine which outcomes are essential to the primary conclusion and which are supplementary.

However, relabeling an essential outcome as exploratory solely because its sample requirement is inconvenient may weaken the scientific purpose.

When several outcomes are indispensable, joint power or multiplicity considerations may be necessary.

See what to do when outcomes require different sample sizes.

Can You Reduce the Number of Study Groups?

Potentially. Removing unnecessary experimental conditions may concentrate recruitment on the comparisons most important to the research question.

For example, a four-arm experiment comparing several feedback strategies may be reduced to a two-arm comparison of AI-generated and conventional feedback.

This may improve power for the remaining contrast when the total sample is fixed.

However, removing groups also removes the ability to answer the corresponding questions.

Researchers should identify which comparisons are indispensable before simplifying the design.

See how the number of groups affects sample size.

Can Changing the Sampling Strategy Solve the Problem?

Changing the sampling strategy may improve access to eligible participants, but it can also alter the scope of inference.

For example, recruiting volunteers from several accessible classes may increase participation compared with a probability sample drawn from a broader student population.

However, the convenience sample may not support the same population generalization.

Similarly, changing from individual randomization to classroom-level randomization may simplify implementation but introduce clustering that increases sample size requirements.

Researchers should evaluate sampling feasibility and inferential validity together.

Recruitment convenience should not be confused with statistical efficiency or representativeness.

Should You Narrow the Population or Expand Recruitment?

These strategies solve different problems.

Narrowing the population may improve conceptual focus and access to participants, but it may limit generalizability.

Expanding recruitment across institutions may increase available participants, but it can introduce additional contextual variation and implementation demands.

For example, a multi-university AI feedback study may require consistent intervention delivery and assessment procedures across sites.

Researchers should determine whether the additional or restricted population remains appropriate for the original research question.

Neither strategy guarantees a smaller statistical requirement.

When Should You Change from Confirmatory to Exploratory Research?

A change may be appropriate when the feasible sample cannot support the intended confirmatory conclusion but can provide useful preliminary evidence.

For example, a smaller study may estimate the direction and uncertainty of an intervention effect, evaluate implementation procedures, or examine recruitment feasibility.

However, exploratory status should be reflected in the analysis and reporting.

Researchers should not describe a study as exploratory while making definitive claims that the intervention is effective or ineffective.

A smaller sample can support a different research objective, but it does not preserve the original power merely through a change in terminology.

Can a Feasibility Study Be More Appropriate Than a Reduced Effectiveness Trial?

Yes, particularly when uncertainty about implementation or recruitment prevents a credible definitive study.

For example, researchers may need to establish whether students complete AI feedback activities, whether the writing assessment is practical, and whether follow-up retention is acceptable.

A feasibility study can provide evidence for planning a subsequent trial.

Its sample size should be justified through the feasibility objectives or desired precision of relevant planning parameters.

However, feasibility evidence should not be interpreted as definitive evidence of treatment effectiveness.

How Should You Compare Different Revision Options?

Researchers should evaluate each proposed change according to whether it preserves the estimand, improves statistical information, changes the population, or alters the intended conclusion.

No option is universally preferable. The appropriate choice depends on the study's scientific purpose, available evidence, and practical constraints.

What If the Available Sample Is Fixed?

When recruitment cannot be expanded, researchers can evaluate the statistical properties of the feasible design.

For hypothesis testing, sensitivity analysis may identify the effect magnitude detectable with specified power.

For estimation, researchers may calculate the anticipated confidence interval width.

These calculations help determine whether the feasible sample can provide useful evidence for the intended objective.

However, a detectable effect should not automatically be treated as the smallest meaningful effect.

The researcher should compare statistical capability with substantive importance before deciding whether to proceed.

How Should Revisions Be Documented?

Researchers should retain a record of the original objective, sample size assumptions, recruitment constraints, and reasons for any revision.

The revised protocol should identify the primary outcome, estimand, analysis, and sample size justification.

If changes occur after recruitment begins, the timing and relationship to available outcome data should be reported.

Changes informed by observed treatment effects raise different concerns from revisions made prospectively using external evidence.

Where registration or ethics approval applies, relevant amendments should follow the applicable requirements.

Transparent documentation helps readers distinguish scientifically justified adaptation from selective analytical decision-making.

How Should You Explain the Final Decision in a Research Paper?

The methodology should explain the final research objective and the calculation used to justify the achieved or planned sample.

If the design was revised because the original sample was unattainable, describe the substantive rationale and any important changes to the estimand.

For example, a researcher might explain that baseline-adjusted analysis was selected prospectively because baseline performance was expected to explain substantial outcome variability.

Alternatively, the researcher may explain that the investigation was redesigned as a feasibility study because the available population could not support a definitive effectiveness trial.

The results should be interpreted according to the final design's capabilities rather than the ambitions of the original proposal.

Choosing Between Revising the Design and Revising the Research Question

The standardized effect corresponding to the meaningful difference is:

Using the conventional normal approximation:

The researcher considers three possible revisions.

Under the simplified baseline-adjustment relationship, a correlation of 0.70 gives 1 − 0.70² = 0.51.

Multiplying the original unrounded per-group requirement by 0.51 gives approximately 128.1 participants per group, or 129 after rounding. The corresponding illustrative total is 258.

This is a simplified approximation, not a definitive ANCOVA sample size calculation. It assumes that the baseline adjustment produces the stated residual variance reduction and that other relevant conditions remain appropriate.

The result is still above the university's capacity of 160 students.

Changing the planning effect to 0.50 produces a feasible calculation, but only by targeting a substantially larger effect than the original minimum meaningful effect.

The researcher therefore has a genuine decision to make.

If detecting the three-point difference is essential, the original confirmatory objective remains infeasible without greater recruitment capacity or a sufficiently efficient alternative design.

If the university cannot expand recruitment, a preliminary investigation of implementation feasibility and effect estimation may be more defensible.

The example illustrates that revising the design can improve efficiency, whereas revising the effect assumption may simply change the question being tested. The two should not be confused.

Common Mistakes When Revising a Study Because of Sample Size Constraints

Which Revision Should You Prioritize When Sample Size Is Unrealistic?

Begin with the scientific objective and evaluate whether the available sample can support it. Prefer changes that improve information quality while preserving the intended estimand, but recognize when a revised objective is necessary.

A defensible revision should have a clear methodological rationale independent of its effect on the calculated sample size.

Document which features of the original question remain unchanged and which have been altered. This distinction helps prevent an efficient-looking study from answering a different question without acknowledging it.

Before Revising a Study, Check What the Change Actually Accomplishes

Frequently Asked Questions About Revising Studies with Unrealistic Sample Sizes

Revise the Research Honestly, Not Merely the Numbers in the Calculation

How to Cite This Guide

This guide is intended to be read, shared, and used in research, teaching, and academic work. If you draw on its ideas, explanations, or other content, please acknowledge the source by citing the guide. Doing so gives appropriate credit and helps your readers locate the original resource.