03 · What You Need to Know
What Pilot Studies Can and Cannot Tell You About the Effect to Expect
What Is a Pilot Study Supposed to Establish?
A pilot study is a smaller investigation conducted to inform the design or implementation of a subsequent study. In intervention research, an external pilot is conducted separately before the definitive study, whereas an internal pilot forms an initial phase of the main study and may contribute data to the final analysis under a prespecified design.
Pilot and feasibility studies often investigate whether recruitment procedures work, participants can complete assessments, an intervention can be delivered as intended, and the proposed data-collection process is manageable.
These questions differ from the primary effectiveness question of a definitive trial. A pilot can demonstrate that a procedure is feasible without providing a reliable estimate of the intervention's true effect.
Eldridge and colleagues (2016) emphasize the distinction between feasibility and pilot studies and the importance of treating pilot studies as preparation for larger investigations rather than simply as underpowered effectiveness trials.
Why Is a Pilot Study's Observed Effect Size Often Unreliable?
An observed effect size is a statistical estimate. Its value varies from sample to sample because participants differ, measurements contain error, and the sample represents only part of the population.
When a pilot has few participants, the sampling variability of the estimated effect can be substantial. A moderate underlying effect might appear large in one pilot and nearly absent in another.
This instability becomes especially consequential when the estimated effect is used to calculate sample size. For many conventional designs, the required sample size is approximately inversely proportional to the square of the assumed standardized effect.
Consequently, an upwardly exaggerated pilot estimate can produce a much smaller sample-size recommendation than the true effect would warrant.
The reasons that pilot effect size estimates fluctuate deserve particular attention when researchers are considering a small preliminary experiment.
Does a Statistically Significant Pilot Result Make Its Effect Size More Reliable?
No. Statistical significance does not eliminate sampling uncertainty.
Indeed, when a small study is only capable of producing statistically significant results for unusually large observed effects, selecting estimates because they achieved significance can introduce substantial exaggeration.
This phenomenon is related to selection bias and the winner's curse: estimates selected for unusually favorable results tend, on average, to overstate the underlying effect.
A pilot result that fails to reach statistical significance is not necessarily evidence that the intervention has no effect either. Most pilots are not designed to provide decisive tests of effectiveness.
Watch Out
Do not select a pilot effect size for the main study merely because the pilot produced a statistically significant result. Selection based on favorable preliminary findings can make the effect-size assumption particularly optimistic.
Why Does Directly Using the Pilot Estimate Create Problems?
Suppose a small pilot produces d = 0.70, while the underlying population effect is closer to d = 0.35. If researchers calculate the main study's sample size using d = 0.70, they may plan a study with substantially less power than intended at d = 0.35.
The calculation itself may be correct for the assumed effect. The problem lies in the uncertain assumption.
Albers and Lakens (2018) examine how effect-size estimation and follow-up selection can bias power analyses based on pilot data. Their work illustrates why treating a noisy preliminary estimate as a known population parameter can be misleading.
This is also why the difference between the expected effect and the smallest effect of interest matters. A preliminary estimate predicts what might occur; it does not independently establish what magnitude would be scientifically or practically important.
What Information From a Pilot Study May Be More Useful?
For many studies, the most valuable pilot outputs concern design parameters and implementation conditions rather than the observed intervention effect.
| Pilot information |
Potential planning use |
Important limitation |
| Outcome standard deviation |
Helps calculate sample size for continuous outcomes. |
May be imprecise in a small pilot and sensitive to the population sampled. |
| Recruitment and eligibility rates |
Helps estimate how many people must be approached. |
May change across sites, seasons, or recruitment procedures. |
| Attrition and missing-data rates |
Informs retention strategies and recruitment allowances. |
Short pilots may not reflect longer follow-up periods. |
| Measurement completion and reliability |
Identifies weaknesses in instruments and assessment procedures. |
Small samples may yield uncertain reliability estimates. |
| Observed intervention effect |
Provides preliminary evidence about plausible effect magnitudes. |
Frequently too imprecise to justify a single planning effect. |
The usefulness of a pilot standard deviation should not be overstated. Browne (1995) and Julious (2005) discuss the role of preliminary information in sample-size planning, including the difficulties associated with small pilot samples.
If outcome variability is unknown, a pilot may help establish a plausible range. Researchers should nevertheless account for uncertainty rather than assuming that the observed standard deviation is exact.
Can a Larger Pilot Produce a More Reliable Effect Estimate?
Generally, increasing the pilot sample size improves the precision of an effect estimate when the design and assumptions remain appropriate. However, the relevant question is whether the resulting precision is sufficient for the intended planning decision.
A pilot with 100 participants may provide more information than one with 20, but it is not automatically large enough to estimate a modest treatment effect precisely.
Reliability also depends on measurement quality, sampling, intervention implementation, and comparability with the definitive study.
At some point, a preliminary study designed primarily to estimate effectiveness may resemble a smaller substantive study rather than a conventional feasibility pilot. Researchers should be clear about which objective they are pursuing.
Should You Use the Pilot Effect Size, Its Confidence Interval, or a Lower Estimate?
Reporting the pilot effect estimate together with a confidence interval makes its uncertainty more visible. A wide interval indicates that multiple underlying effect magnitudes remain compatible with the data under the assumed model.
However, automatically substituting the lower confidence limit into a sample-size formula is not a universal solution. The lower limit may be negative or close to zero, potentially producing an impractical or undefined planning target for a particular calculation.
A more defensible procedure is to identify plausible effects using the pilot, prior literature, and substantive considerations, then examine how sample-size requirements change across those values.
For example, a researcher might calculate sample sizes at d = 0.25, 0.35, and 0.45 rather than relying exclusively on an observed pilot value of d = 0.45.
Such a range of plausible assumptions reveals how strongly the design depends on the uncertain effect.
Can the Smallest Effect Size of Interest Replace a Pilot Estimate?
Sometimes. If researchers can establish a smallest effect size of interest (SESOI), that threshold may provide a more meaningful target for a conventional superiority power analysis than an unstable pilot estimate.
For example, a pilot might produce d = 0.65, but the research team might determine that d = 0.30 is the smallest improvement worth detecting. If the study must be adequately powered for d = 0.30, using the larger pilot estimate would not meet that objective.
However, a SESOI does not automatically solve every planning problem. It must be substantively justified, and the sample-size method must match the actual hypothesis.
What About Bayesian Approaches to Pilot Information?
Bayesian planning can incorporate prior information and uncertainty about the effect rather than treating a single estimate as certain.
For example, researchers may specify a probability distribution representing plausible effects, update that distribution using pilot data, and evaluate the expected operating characteristics of alternative sample sizes.
This approach can be useful when credible prior evidence exists, but it requires transparent prior assumptions and an appropriate model. Bayesian methods do not automatically correct poor pilot data or unsupported prior beliefs.
What Is Different About an Internal Pilot Study?
An internal pilot is embedded within the main study and may support prespecified sample-size reassessment.
For example, a trial may initially recruit participants, estimate a nuisance parameter such as outcome variance, and revise the final sample size according to a predefined procedure.
Blinded sample-size reassessment based on nuisance parameters can sometimes be implemented without using interim treatment-effect estimates. Unblinded reassessment may require additional safeguards to preserve the intended statistical operating characteristics.
Researchers should distinguish such planned adaptive procedures from changing the sample size informally after inspecting favorable or unfavorable interim results.
