03 · What You Need to Know
How to Make Defensible Sample-Size Assumptions Under Uncertainty
Why Does Effect Size Matter in Sample-Size Planning?
In conventional power-based sample-size calculations, the effect size represents the magnitude of the difference, association, or other effect that the study is designed to detect under specified assumptions.
For example, a researcher comparing two teaching approaches might express the anticipated difference in examination scores as a raw mean difference or a standardized mean difference. A researcher investigating an association might use a correlation coefficient, while a clinical researcher might specify a risk difference or hazard ratio.
Other planning inputs typically include the significance level, desired statistical power, outcome variability, allocation ratio, and statistical test. Which inputs are necessary depends on the design and analysis.
When other assumptions remain unchanged, smaller effects generally require larger samples to achieve the same statistical power. This is why the effect-size input deserves substantive justification rather than being treated as an administrative requirement of statistical software.
Distinguish the Expected Effect From the Effect Worth Detecting
Two different questions often become conflated during sample-size planning:
Expected effect size
The effect magnitude considered plausible or likely based on prior evidence, theory, or other justified assumptions.
Smallest effect size of interest
The smallest effect that would be sufficiently meaningful to influence a substantive conclusion or decision in the research context.
These quantities need not be identical. You might expect an intervention to improve performance by eight points but regard a four-point improvement as educationally meaningful.
For a conventional power calculation, you can often use the smallest effect worth detecting rather than the expected effect, provided that the choice is consistent with the study's inferential objective.
This does not mean that a meaningful threshold can always be established easily. It may require disciplinary judgment, consultation with relevant stakeholders, or consideration of the intervention's costs and consequences.
Start With the Effect in Its Original Measurement Units
Researchers sometimes begin by searching for a standardized effect size such as Cohen's d. Yet a difference expressed in the original outcome units may be easier to interpret and defend.
Suppose a new instructional intervention is intended to improve examination scores measured on a 100-point scale. Instead of immediately selecting d = 0.50, ask what improvement would justify adopting the intervention.
Would a two-point improvement matter? Would five points change a meaningful educational decision? Would a ten-point gain justify substantial additional instructional time?
These questions connect sample-size planning to the practical purpose of the research.
If a standardized effect is required, the meaningful raw difference can be divided by an appropriate planning estimate of the outcome standard deviation. However, uncertainty about that standard deviation must also be addressed. A defensible meaningful difference does not eliminate uncertainty about variability.
Use Previous Research Without Treating Its Estimates as Certain
Existing studies may provide useful information even when none offers a directly transferable effect estimate.
Consider whether previous research reports:
- Effects on the same outcome using comparable measurement instruments.
- Effects in populations with reasonably similar characteristics.
- Interventions with comparable mechanisms, intensity, and implementation conditions.
- Confidence intervals that reveal uncertainty around reported effects.
- Outcome standard deviations, baseline rates, or other planning parameters.
A systematic review or meta-analysis may provide a more informative starting point than an isolated statistically significant result. Nevertheless, pooled estimates can also conceal important heterogeneity.
If the most relevant study used a substantially different population, its effect estimate should be treated as a provisional assumption rather than automatically transferred. The suitability of effect sizes from different populations depends on whether the mechanisms and conditions producing the effect are likely to be comparable.
It is also important to avoid choosing the largest published effect merely because it produces an affordable sample size. Published estimates, especially from small studies, may be imprecise or affected by selective reporting.
Use Sensitivity Analysis Instead of Pretending There Is One Certain Effect
When uncertainty remains, calculate the required sample size for several plausible effect sizes.
For example, a researcher might examine standardized mean differences of 0.20, 0.30, and 0.40, provided there is a substantive or empirical reason to consider that range plausible.
These values should be interpreted as planning scenarios, not as claims that the true effect must lie within the selected interval.
A range of plausible planning assumptions can reveal how sensitive recruitment requirements are to uncertain inputs. A more formal sample-size sensitivity analysis can also vary outcome variability, attrition, and other influential parameters.
The resulting calculations make trade-offs visible. If the sample size needed to detect a small meaningful effect is infeasible, that limitation should influence the study design rather than be concealed by choosing a larger assumed effect.
Why Conventional Small, Medium, and Large Effects Are Not Automatic Solutions
Cohen's conventional benchmarks can help researchers understand standardized magnitudes in some contexts, but they are not universal definitions of substantive importance.
A standardized effect of 0.20 might represent a valuable improvement in a large-scale educational program, while a standardized effect of 0.50 might be insufficient to justify an expensive intervention with substantial implementation burdens.
What matters is the research context, the outcome, and the consequences of the effect.
Using a conventional benchmark solely because a calculator requires a value produces a numerical answer, but it does not establish that the resulting sample size is appropriate.
Can a Pilot Study Solve the Problem?
A pilot study may help estimate recruitment rates, assess procedures, test instruments, and identify practical difficulties. It may also provide preliminary information about outcome variability.
However, the observed treatment effect from a small pilot is often highly uncertain. Designing the main study around that single estimate can produce an inadequately powered study, especially if the pilot effect is unusually large.
The question is therefore not simply whether a pilot study can provide an effect-size estimate, but whether that estimate is sufficiently precise and relevant to justify its use.
For many external pilot studies, feasibility parameters are more defensible primary targets than the preliminary intervention effect.
Consider Precision-Based Planning When Estimation Is the Main Objective
Not every quantitative study needs to be organized around detecting an effect through a significance test.
If the primary objective is to estimate a population mean, prevalence, correlation, or other parameter with acceptable uncertainty, sample size may be determined by a desired confidence-interval width.
For example, a researcher estimating the proportion of students experiencing a particular learning barrier might specify a desired margin of error rather than a hypothetical effect size.
The distinction between precision-based and power-based sample-size planning is particularly important when the research question is descriptive rather than confirmatory.
Precision-based planning still requires assumptions, such as anticipated variability or prevalence. It does not make uncertainty disappear, but it aligns the calculation with a different inferential objective.
What If Neither the Expected Effect Nor a Meaningful Threshold Can Be Justified?
Sometimes the available evidence is too limited to support a credible numerical assumption, and stakeholders cannot yet identify a meaningful effect threshold.
In that situation, researchers should reconsider what the proposed study can reasonably accomplish.
Options may include an explicitly exploratory study, an estimation-focused investigation, a feasibility study, or a staged research program that develops the evidence needed for a later confirmatory design.
A resource-constrained sample can still generate useful evidence, but the achievable power or precision should be reported transparently. It should not be presented as adequate merely because the available number of participants was entered into a calculator.
Watch Out
Do not calculate the sample size using a convenient large effect and subsequently describe the study as adequately powered for smaller effects. Statistical power is conditional on the effect magnitude and other assumptions used in the calculation.
04 · A Practical Example
Planning an Educational Intervention Without a Reliable Effect Estimate
Hypothetical Example
Evaluating a New Online Learning Strategy
A researcher plans a two-group randomized study comparing a new online learning strategy with conventional instruction. The primary outcome is a continuous achievement score.
There is no closely comparable previous intervention study. The researcher therefore examines several standardized effect sizes rather than claiming to know the likely effect.
For illustration, assume an independent two-sample, two-sided t-test with equal group sizes, a significance level of 0.05, and target power of 80%.
| Assumed standardized effect (d) |
Approximate participants per group |
Approximate total |
| 0.20 |
394 |
788 |
| 0.30 |
176 |
352 |
| 0.40 |
100 |
200 |
| 0.50 |
64 |
128 |
These rounded figures illustrate conventional power calculations under the stated assumptions. Exact requirements can vary with the calculation method, rounding convention, and design specification. They exclude attrition and other design adjustments.
Suppose educational stakeholders determine that an effect equivalent to d = 0.30 is the smallest worthwhile improvement, while an effect of 0.40 remains plausible.
The researcher could plan for approximately 352 analyzable participants to target 80% power at d = 0.30, then separately account for expected unusable data or attrition.
If only 200 participants are realistically available, the study cannot simply claim the same power for d = 0.30. The researcher must evaluate the achievable power, reconsider the design, or revise the study's inferential ambitions.
The essential lesson is that uncertainty should be reflected in the planning analysis, not concealed inside a single unjustified number.
06 · What This Means for You
Choose a Planning Strategy That Matches Your Research Objective
Begin with the purpose of the study rather than the requirements of a particular calculator. The most appropriate planning strategy depends on whether you are testing a hypothesis, estimating a parameter, or establishing feasibility.
A simple decision framework
If a meaningful minimum effect can be specified
Consider powering the study for that threshold, while evaluating whether the required sample is feasible and the assumed variability is defensible.
If relevant prior studies exist but their estimates differ
Examine study comparability, uncertainty, and heterogeneity; calculate sample sizes under multiple plausible scenarios.
If the objective is primarily estimation
Specify acceptable confidence-interval precision and use a sample-size method appropriate for the parameter being estimated.
If evidence and resources are both limited
Consider a feasibility or exploratory design, and report its intended contribution and inferential limitations explicitly.
Document why each numerical assumption was selected. A transparent justification should identify the primary outcome, effect metric, significance level, power target, relevant variability assumptions, and sensitivity of the result to uncertainty.
For clustered, repeated-measures, or otherwise complex designs, use a calculation that reflects the actual analysis. A sample-size result for independent observations may be inappropriate when participants are nested within classes or institutions.
