What Is Effect Size, and Why Does the Expected Effect Affect Sample Size?

Why Does Detecting a Small Improvement Require More Participants Than Detecting a Large One?

You are designing an experiment to investigate whether AI-generated feedback improves university students' academic writing performance. Your primary outcome is a writing assessment scored from 0 to 100.

One previous study suggests that a similar intervention might improve scores by approximately eight points. Another reports a much smaller difference. Your colleagues recommend using a large expected effect because it would reduce the calculated sample size.

Would choosing a larger effect make the study more feasible? Mathematically, it often would. But would the resulting calculation still provide a credible basis for the research?

Effect size is among the most consequential assumptions in statistical power analysis. Small changes in the effect used for planning can produce substantial differences in the number of participants required.

The methodological challenge is to distinguish the effect researchers expect to observe from the effect that would be meaningful to detect, and to avoid treating effect size as a convenient numerical input chosen to produce an affordable study.

Smaller Effects Generally Require Larger Samples to Detect Reliably

Understanding Effect Size and Its Role in Sample Size Planning

What Exactly Is Effect Size?

Effect size describes the magnitude of a quantity of interest, such as a difference between groups, a relationship between variables, or a change associated with an intervention.

For example, an educational intervention may increase mean writing scores by five points, reduce completion time by three minutes, or produce a difference in the probability of successfully completing an assessment.

These are different forms of effect, measured on different scales.

Effect size is not a single universal statistic. Its definition depends on the research question, outcome, and statistical model.

Researchers should identify the estimand, meaning the quantity they intend to estimate or test, before choosing an effect size measure.

What Is the Difference Between Raw and Standardized Effect Size?

A raw effect is expressed in the original units of the outcome. A standardized effect expresses the magnitude relative to a measure of variability or another reference scale.

Raw effects may be easier to interpret substantively because they retain the original measurement scale.

Standardized effects may facilitate comparisons across outcomes measured differently, but their interpretation depends on the variability and population used for standardization.

Neither is inherently superior for every research purpose.

How Is Cohen's d Calculated?

For a comparison of two independent group means, Cohen's d commonly expresses the mean difference relative to a pooled within-group standard deviation.

For prospective planning, researchers generally use a hypothesized population standardized difference rather than a sample estimate already observed in the proposed study.

Other designs, including paired comparisons and repeated-measures studies, may require different standardization conventions.

Are Cohen's Small, Medium, and Large Effect Categories Universal?

No. Cohen's commonly cited conventions of approximately 0.20, 0.50, and 0.80 for standardized mean differences were intended as general reference points, not universal definitions of scientific importance.

A standardized effect of 0.20 may be educationally meaningful when an intervention is inexpensive, scalable, and affects many learners.

Conversely, an effect of 0.50 may be insufficient to justify a costly or burdensome intervention.

Interpretation should consider the outcome, educational context, consequences, and relevant evidence.

Researchers should avoid selecting a conventional category without explaining why the corresponding effect matters for the particular study.

Why Does a Smaller Effect Require a Larger Sample?

Statistical tests evaluate whether the observed data provide sufficient evidence against a null hypothesis under a specified model.

When the true difference between groups is small relative to outcome variability, the sampling distributions under the null and alternative hypotheses overlap substantially.

A larger sample generally reduces the standard error of the estimated difference, making a specified small effect easier to distinguish from random variation.

For many common designs, the standard error decreases approximately in proportion to the inverse square root of sample size.

Consequently, detecting progressively smaller effects requires disproportionately more observations.

The exact relationship depends on the outcome distribution, statistical test, allocation, and study design.

What Is the Mathematical Relationship Between Effect Size and Sample Size?

For two independent groups with equal allocation, approximately equal outcome variances, and a two-sided test, a common normal approximation is:

When the standardized effect is defined as d = Δ / σ, the expression becomes:

What Happens When the Expected Effect Is Halved?

Because the effect appears squared in the denominator, halving the effect approximately multiplies the required sample by four under the stated assumptions.

For example, detecting a standardized effect of 0.25 generally requires approximately four times the sample needed to detect an effect of 0.50 with the same alpha and power.

Doubling the assumed effect has the opposite approximate relationship, reducing the requirement to one quarter.

This explains why sample size calculations can be highly sensitive to effect assumptions.

However, researchers should not use this relationship to justify increasing the assumed effect merely because the resulting sample is easier to recruit.

What Is an Expected Effect Size?

An expected effect size is a planning assumption about the magnitude of an effect that might occur under the proposed study conditions.

It may be informed by previous research, theory, pilot data, or relevant contextual evidence.

However, an expected effect is uncertain. It is not a promise that the study will observe that magnitude.

For example, previous studies of AI-generated feedback may differ in participant characteristics, intervention duration, assessment procedures, and outcome measures.

A pooled effect from earlier research may therefore require careful evaluation before being transferred to a new setting.

Researchers should distinguish an empirical expectation from the effect they consider important enough to detect.

What Is the Smallest Effect Size of Interest?

The smallest effect size of interest, sometimes abbreviated SESOI, is the smallest effect considered substantively meaningful for the research question.

For example, a university may consider a three-point improvement in a validated writing assessment educationally worthwhile if the intervention requires minimal resources.

That threshold should be justified through educational significance, decision consequences, prior evidence, or stakeholder-informed criteria.

It should not be selected solely because it produces a feasible sample size.

Planning for the smallest meaningful effect can be more defensible than planning for an optimistic effect that may not occur.

However, the appropriate approach depends on the study's inferential objective and the uncertainty surrounding plausible effects.

Is the Expected Effect the Same as the Minimum Important Effect?

No. The expected effect describes what researchers anticipate, while the minimum important effect describes what would matter substantively.

Suppose previous studies suggest that AI feedback might improve writing scores by six points, but the university would consider a three-point improvement worthwhile.

A study powered only to detect six points may have insufficient power to detect the smaller but still meaningful three-point effect.

Researchers should therefore examine whether the planning effect reflects the actual scientific or policy decision.

These distinctions are central to statistical power in study design.

Can Previous Studies Provide a Reliable Effect Size?

Previous studies can provide useful evidence, but their estimates should not be adopted automatically.

Researchers should consider study design, population, intervention characteristics, outcome measurement, risk of bias, and precision.

For example, an effect estimated in a short laboratory experiment may not generalize directly to semester-long classroom implementation.

Small studies may also produce unstable effect estimates, particularly when published findings are selectively significant.

Meta-analytic estimates may be informative, but their relevance depends on the comparability and quality of included studies.

When evidence is uncertain, sensitivity calculations across several plausible effects may be preferable to reliance on a single estimate.

Should You Use the Effect Size from a Pilot Study?

Pilot studies may provide preliminary information about outcome variability, recruitment, missingness, and procedures.

However, estimated intervention effects from small pilots can be highly unstable.

For example, a pilot involving 12 participants per group might produce an apparently large effect because of random variation.

Using that estimate directly for a definitive trial could substantially underestimate the required sample.

Researchers should evaluate the uncertainty of pilot effect estimates and consider whether external evidence or a substantively justified threshold provides a stronger planning basis.

Pilot effect estimates should not be treated as reliable merely because they are available.

How Does Outcome Variability Interact with Effect Size?

A raw difference of five points may be relatively large when scores have a standard deviation of five points, but relatively small when the standard deviation is twenty points.

The standardized difference therefore depends on both the mean difference and outcome variability.

For example, five points divided by a standard deviation of ten produces d = 0.50, while five points divided by twenty produces d = 0.25.

Under otherwise identical independent-means planning assumptions, the second scenario requires approximately four times the sample.

Researchers should therefore justify both the raw effect and variability assumptions.

See how outcome variability affects sample size for further discussion.

Do All Effect Size Measures Affect Sample Size in the Same Way?

No. Different outcomes and statistical models use different effect measures.

A comparison of means may use a raw or standardized mean difference.

A binary outcome comparison may use a risk difference, risk ratio, or odds ratio, together with assumptions about baseline event probability.

Correlation studies may use an anticipated correlation coefficient, while survival analyses may involve hazard ratios and event rates.

The sample size relationship depends on the effect measure and other parameters.

Researchers should use calculations appropriate to the actual estimand and statistical analysis rather than substitute a familiar standardized effect into an unrelated formula.

Does Effect Size Matter in Descriptive Studies?

Descriptive studies may not have a treatment effect or group difference to detect.

For example, a survey estimating the prevalence of weekly AI use may be planned according to a desired margin of error rather than a standardized mean difference.

In that setting, the anticipated proportion and desired precision are more relevant.

Effect size becomes central when the objective concerns detecting or estimating a particular difference, association, or relationship.

Not every sample size problem should therefore be approached through Cohen's d.

What If the Expected Effect Is Unknown?

Researchers should avoid inventing an effect merely to complete a calculation.

Possible responses include identifying a smallest meaningful effect, examining a plausible range, obtaining preliminary information, or reconsidering the study's objective.

If no defensible effect can be specified, an exploratory or feasibility investigation may be more appropriate than a definitive confirmatory trial.

Simulation or sensitivity analysis may help examine the consequences of uncertain assumptions.

The approaches discussed in estimating sample size without a formal calculation are relevant when essential planning information remains unavailable.

How Should Effect Size Assumptions Be Reported?

The methodology should identify the effect measure, numerical planning value, source or substantive rationale, and associated variability assumptions.

Researchers should state whether the effect represents an expected effect, a minimum meaningful effect, or another planning scenario.

If sensitivity analysis was conducted, report the range examined and how requirements changed.

The final results should report the observed effect estimate and its uncertainty rather than imply that the planning value was known to be correct.

Transparent reporting allows readers to evaluate whether the sample size justification was scientifically credible.

How Different Effect Assumptions Change the Required Sample

The researcher examines three hypothetical meaningful mean differences rather than selecting a single optimistic effect.

The values use the normal approximation n ≈ 2(1.96 + 0.84)² / d², with each per-group result rounded upward. Exact noncentral t calculations may differ slightly.

Planning for a 2.5-point difference requires substantially more participants than planning for a ten-point difference.

However, the researcher cannot justify using ten points merely because recruiting 32 students is easier than recruiting 502.

If a five-point improvement is the smallest educationally meaningful effect, the study should consider whether it can recruit approximately 126 usable observations under the stated assumptions, plus any appropriate adjustment for missing data.

If that target is infeasible, the researcher should reconsider the design or research objective rather than inflate the assumed effect.

The example demonstrates why the effect size used in planning must be substantively defensible.

Common Misunderstandings About Effect Size and Sample Size

How to Choose an Effect Size for Sample Size Planning

Begin by defining the effect that matters for the research question and identifying the evidence available to justify its magnitude.

In the methodology section, state the effect measure, planning value, rationale, and assumptions about outcome variability.

Explain whether the value represents an expected effect or a minimum meaningful effect.

Effect size assumptions should reflect the scientific question, not the number of participants researchers would prefer to recruit.

Before Using an Effect Size in a Sample Size Calculation, Verify Its Basis

Frequently Asked Questions About Effect Size and Sample Size

The Effect You Aim to Detect Determines How Much Evidence You Need

How to Cite This Guide

This guide is intended to be read, shared, and used in research, teaching, and academic work. If you draw on its ideas, explanations, or other content, please acknowledge the source by citing the guide. Doing so gives appropriate credit and helps your readers locate the original resource.