Why Are Pilot Study Effect Size Estimates Often Unstable?

Why Can a Pilot Study Report a Large Effect That Disappears in a Larger Study?

A pilot study involving 24 participants reports a standardized effect size of 0.80. Encouraged by the result, the researchers conduct a larger investigation. This time, the estimated effect is only 0.25.

Did the intervention stop working? Was the pilot poorly conducted? Or did the larger study contradict the earlier finding?

Not necessarily. Even when two studies investigate the same population effect using comparable methods, their estimates can differ substantially because of sampling variability. Small pilot studies are particularly susceptible to this fluctuation, and certain analytical and selection practices can make the problem worse.

Small Samples Produce Noisy Effect Estimates, and Selection Can Exaggerate Them

The Statistical Reasons Pilot Effect Sizes Can Change Dramatically

Sampling Error: Different Participants Produce Different Estimates

An effect size calculated from sample data is an estimate of an underlying population quantity. If researchers repeatedly selected independent samples from the same population, they would generally obtain different effect estimates.

This variation is called sampling variability. It arises because each sample contains a different combination of observations.

For example, one pilot sample may happen to contain intervention participants who respond particularly well. Another sample may contain participants whose responses are closer to those of the comparison group.

Neither outcome necessarily indicates that the population effect has changed.

With small samples, individual observations can have a relatively large influence on the estimated group means and other statistics. Increasing sample size generally reduces sampling uncertainty when the observations provide genuinely additional information and the statistical model is appropriate.

Why Does Sample Size Affect the Precision of an Effect Estimate?

Consider an independent comparison of two population means with a common standard deviation. Under the usual assumptions, the standard error of the unstandardized difference between two sample means is:

This relationship explains why larger samples generally yield more precise mean-difference estimates. However, standardized effect sizes have additional uncertainty because the standard deviation used in standardization must also be estimated.

Why Is Cohen's d Particularly Variable in Small Samples?

Cohen's d is commonly calculated by dividing the difference between two sample means by a pooled sample standard deviation.

Its value therefore depends on two estimated quantities: the numerator representing the observed mean difference and the denominator representing observed variability.

Both fluctuate from sample to sample.

If a small pilot happens to produce a relatively large mean difference and a relatively small pooled standard deviation, the resulting standardized effect can be especially large.

Conversely, a smaller observed mean difference or larger pooled standard deviation can produce a much smaller value.

Small-sample bias is another consideration. The conventional sample Cohen's d is not an exactly unbiased estimator of the population standardized mean difference under the usual normal-model assumptions. Hedges' g applies a correction intended to reduce that bias.

However, bias correction does not remove sampling variability. A corrected estimate can remain highly imprecise when the pilot contains few observations.

What Is the Difference Between Bias and Instability?

These concepts are related but distinct.

An estimator can be unbiased yet highly variable. It can also be relatively precise but systematically biased.

This distinction matters because an individual pilot estimate that overstates the population effect is not, by itself, proof of a biased estimator. The overstatement may simply reflect random sampling variation.

Selection practices, however, can create systematic distortions even when the original estimation procedure is otherwise reasonable.

Why Are Large Pilot Effects Particularly Vulnerable to the Winner's Curse?

The winner's curse describes a selection-related tendency for unusually favorable observed estimates to overstate their underlying effects.

Suppose researchers conduct several small preliminary studies but proceed only with the intervention producing the largest observed improvement.

The selected intervention may genuinely be the most effective. Nevertheless, its observed estimate is also likely to reflect favorable sampling variation, particularly when the estimates are noisy.

The same problem can arise when researchers pursue only pilots that reach statistical significance.

In a small study with low power, statistically significant results may disproportionately involve observed effects that are larger than the underlying population effect.

Kraemer and colleagues (2006) warned that using pilot studies to guide power calculations can result in worthwhile studies being abandoned while selected follow-up studies are inadequately powered.

The implication is not that large pilot effects are always false. Rather, unusually large effects selected from noisy preliminary results require cautious interpretation.

What Is Follow-up Bias?

Albers and Lakens (2018) describe follow-up bias as a problem that can arise when researchers decide whether to conduct a larger study based on the sample size implied by the pilot estimate.

Consider two pilots investigating interventions with the same underlying population effect.

The first happens to estimate a large effect. Its power calculation suggests that a relatively small, affordable main study will be sufficient.

The second happens to estimate a smaller effect. Its calculation suggests that the main study will require many more participants, making it appear impractical.

If only the first pilot is followed up, the research process preferentially advances studies with favorable effect-size estimation errors.

This selection can systematically undermine the achieved power of the studies that proceed, even without conventional publication bias.

Why Can the Same Pilot Produce Different Results Under Different Effect-Size Measures?

Effect-size measures are not interchangeable, and some estimators have different small-sample properties.

For example, analysis of variance may report eta-squared, partial eta-squared, omega-squared, or other measures. Their definitions, interpretations, and bias characteristics differ.

Albers and Lakens (2018) specifically examine how choosing eta-squared rather than alternative estimators can affect sample-size calculations based on preliminary data.

Eta-squared can overestimate the corresponding population explained-variance quantity in small samples under relevant conditions. Omega-squared and epsilon-squared incorporate corrections that may reduce bias in particular ANOVA settings.

Such corrections should be chosen according to the statistical model and target estimand. They are not universally interchangeable, and no corrected estimator can compensate for inadequate information in a very small sample.

How Do Outliers and Measurement Error Affect Pilot Effect Sizes?

In a small sample, one unusually high or low observation may materially change the estimated mean difference or standard deviation.

Measurement error can also influence observed effect sizes. Random measurement error in an outcome may increase unexplained variability and reduce the apparent standardized effect under some conditions.

Other measurement problems, including differential assessment procedures or systematic bias, may distort the estimate in less predictable ways.

Researchers should investigate unusual observations and measurement quality using prespecified or transparent procedures. Removing observations solely because they make an effect smaller is not an acceptable solution.

It is also important to distinguish statistical instability from changes in the underlying research context. A pilot and a main study may genuinely estimate different effects if their populations, intervention delivery, follow-up periods, or outcome measures differ.

Does Regression to the Mean Explain Why Pilot Effects Often Become Smaller?

Regression to the mean is relevant when observations or estimates are selected because they are unusually extreme.

If a pilot is selected for follow-up because its observed effect is exceptionally large, a subsequent independent estimate will often be less extreme, assuming the underlying effect has not changed and the initial estimate contained sampling error.

This is a statistical consequence of selecting noisy extreme observations, not evidence that the intervention necessarily became less effective.

However, a smaller follow-up effect should not automatically be attributed to regression to the mean. Differences in implementation, study populations, measurement, and analytical procedures may also explain discrepancies.

Why Do Confidence Intervals Matter More Than a Single Pilot Estimate?

A point estimate gives one numerical summary of the observed data. An appropriate confidence interval communicates uncertainty under the assumed statistical model.

For instance, an observed standardized mean difference of d = 0.60 may have a wide confidence interval in a small pilot. The data may remain compatible with effects that are substantially smaller or larger than the point estimate.

Confidence intervals should not be interpreted as assigning a 95% probability that a fixed population effect lies inside the particular observed interval under conventional frequentist inference. Their nominal coverage refers to the long-run behavior of the interval procedure.

For planning purposes, the important observation is that a wide interval reveals limited information about the population effect.

Researchers considering using a pilot estimate for main-study planning should examine its uncertainty rather than treating its reported value as definitive.

Can Increasing the Pilot Sample Size Solve the Problem?

A larger pilot generally provides more precise estimates, assuming its design is appropriate. Yet increasing the pilot sample does not automatically eliminate selection bias, measurement problems, or differences between the pilot and definitive study.

Moreover, a pilot designed primarily to assess feasibility may not need a large sample capable of estimating the treatment effect precisely.

Leon, Davis, and Kraemer (2011) argue that pilot studies should principally inform feasibility and implementation decisions rather than serve as small-scale effectiveness tests.

When precise effect estimation is itself essential, researchers should plan the preliminary study explicitly for that estimation objective instead of assuming that a conventional small pilot will suffice.

How Should Unstable Pilot Effects Be Handled in Sample-Size Planning?

Researchers can reduce dependence on an unstable pilot estimate by using external evidence, a substantively justified effect threshold, and explicit uncertainty analysis.

If the expected effect size remains unknown, this uncertainty should be acknowledged rather than replaced with unwarranted numerical precision.

A smallest effect size of interest may provide a defensible planning target when researchers can establish what magnitude would matter.

Alternatively, a range of plausible effect assumptions can show how sample-size requirements vary across scenarios.

These approaches do not eliminate uncertainty. They make its consequences visible and help researchers avoid allowing one noisy preliminary estimate to determine the entire study design.

How Sampling Variation Can Make an Ordinary Effect Look Remarkably Large

The example does not imply that either pilot estimate is impossible or that every small pilot will produce such extreme values. It illustrates how an uncertain planning effect can produce dramatically different sample-size recommendations because of the nonlinear relationship between effect magnitude and required sample size.

In a real study, exact or simulation-based calculations may be preferable, particularly when sample sizes are small or the intended analysis departs from the simple independent-groups model.

Misconceptions About Why Pilot Effect Sizes Change

Interpret Pilot Effect Sizes as Uncertain Estimates, Not Fixed Design Parameters

The practical implication is to separate the value observed in a pilot from the magnitude your definitive study needs to investigate.

Report the preliminary estimate transparently, examine its uncertainty, and assess whether the pilot was sufficiently informative for the purpose you intend to assign to it.

A sample-size sensitivity analysis can help demonstrate whether the proposed main study remains adequately powered across a defensible range of effects.

Researchers should also be transparent about how preliminary findings influenced the decision to proceed. When the decision depended on the observed effect, that selection process may matter for interpreting subsequent results.

Before Interpreting or Reusing a Pilot Effect Size

Frequently Asked Questions About Unstable Pilot Effect Sizes

Instability Is a Statistical Limitation, Not Necessarily a Research Failure

How to Cite This Guide

This guide is intended to be read, shared, and used in research, teaching, and academic work. If you draw on its ideas, explanations, or other content, please acknowledge the source by citing the guide. Doing so gives appropriate credit and helps your readers locate the original resource.