03 · What You Need to Know
Understanding What Pilot Results Can and Cannot Establish
What Is the Main Purpose of Interpreting Pilot Findings?
The principal purpose is to inform decisions about the proposed definitive study.
A pilot may help researchers determine whether recruitment procedures work, whether participants can complete the intervention, whether outcome assessments are practical, and whether data management procedures function as intended.
These findings may support proceeding with the original design, making modifications, conducting additional feasibility work, or deciding not to proceed.
The appropriate interpretation should therefore begin with the pilot's objectives rather than the most statistically striking result.
For example, if the pilot was designed to assess retention, its central finding concerns the proportion of participants completing follow-up and the implications for the definitive study.
An observed difference in academic achievement may be interesting, but it should not displace the feasibility objective.
What Are Useful Pilot Study Findings?
Useful findings provide evidence relevant to decisions about research implementation.
They may identify procedures that function adequately, difficulties that require modification, or uncertainties that remain unresolved.
| Pilot finding |
Potentially supported conclusion |
Conclusion requiring caution |
| Recruitment rate |
Whether recruitment appears workable under the observed conditions |
That the same rate will necessarily occur in the definitive study |
| Retention rate |
Whether follow-up procedures require modification |
That long-term retention is guaranteed |
| Intervention adherence |
Whether participants can engage with planned activities |
That adherence proves intervention effectiveness |
| Data completeness |
Whether assessment procedures produce sufficiently complete records |
That complete data establish measurement validity |
| Implementation fidelity |
Whether intervention components were delivered as planned |
That successful delivery demonstrates beneficial outcomes |
| Preliminary outcome differences |
Descriptive information that may inform further planning |
Definitive effectiveness or equivalence claims |
| Participant acceptability |
Whether selected procedures or interventions appear acceptable to those studied |
That the intervention will be acceptable to every relevant population |
The distinction is between evidence about feasibility under observed conditions and stronger claims that require a different design or greater precision.
Why Should Pilot Findings Be Linked to Prespecified Objectives?
Prespecified objectives help prevent researchers from interpreting whichever results appear most favorable as the main contribution.
For example, a pilot designed to assess recruitment feasibility should report recruitment outcomes even if preliminary intervention results attract more attention.
Researchers should define what they intend to measure and why the information matters for the definitive study.
When unexpected issues arise, they can be reported as additional findings, but the distinction between planned and exploratory observations should remain clear.
This approach supports transparent interpretation without requiring researchers to ignore genuinely important unanticipated problems.
How Should Recruitment Findings Be Interpreted?
Recruitment findings may help estimate how many participants can be enrolled over a specified period and identify barriers to participation.
For example, a pilot may show that a recruitment procedure yields fewer eligible participants than anticipated.
The researcher can use this information to reconsider recruitment sites, eligibility criteria, staffing, or the duration of the definitive study.
However, recruitment rates are context-dependent. They may change when the study expands to different institutions, participant groups, or recruitment periods.
A pilot recruitment estimate should therefore be interpreted with attention to its precision and the conditions under which it was obtained.
How Should Retention Findings Be Interpreted?
Retention findings indicate the extent to which enrolled participants remain available for specified follow-up activities.
Researchers should define the denominator and follow-up point clearly.
For example, if 27 of 30 enrolled participants complete the final assessment, the observed completion proportion is 90%.
However, this estimate is based on a small sample and does not guarantee the same retention rate in the definitive study.
Researchers should examine why participants withdrew or missed assessments, whether missingness was associated with particular circumstances, and whether follow-up procedures require modification.
Retention should not be interpreted solely through a percentage without considering its context and uncertainty.
How Should Data Completeness Be Interpreted?
Data completeness concerns whether the intended observations or measurements are available.
For example, a pilot may show that most participants complete a questionnaire but frequently omit particular items.
This may suggest that the questionnaire is too long, that some items are unclear, or that the assessment conditions are unsuitable.
However, the missing responses may have other explanations, and researchers should investigate them rather than assuming one cause.
High completeness does not automatically establish measurement validity. Participants may complete every item while misunderstanding the questions.
Completeness is therefore a feasibility indicator, not a substitute for evaluating measurement quality.
How Should Intervention Fidelity Be Interpreted?
Intervention fidelity concerns the extent to which planned intervention components were delivered or implemented as intended.
For example, a pilot of an AI-supported formative feedback system may examine whether teachers complete training, use the system during specified activities, and provide feedback according to the protocol.
High fidelity may indicate that the intervention can be implemented under the observed conditions.
However, successful implementation does not establish that the intervention improves learning outcomes.
Similarly, low fidelity does not necessarily mean that the underlying intervention is ineffective. It may indicate that delivery procedures require modification.
Researchers should distinguish implementation problems from conclusions about intervention effects.
What Can Acceptability Findings Tell You?
Acceptability findings may indicate how participants or implementers experience the proposed intervention and research procedures.
Researchers may examine perceived burden, appropriateness, convenience, or willingness to participate.
For example, teachers may report that an intervention is educationally useful but difficult to implement within existing workloads.
Such findings can inform revisions to training, scheduling, or intervention delivery.
However, favorable acceptability does not establish effectiveness.
Acceptability may also vary across populations, settings, and stages of implementation.
Why Are Pilot Effectiveness Estimates Often Unreliable?
Pilot studies are commonly too small to estimate intervention effects with sufficient precision for definitive conclusions.
Small samples can produce unstable effect estimates that vary substantially from the effects observed in larger studies.
For example, a pilot may produce a large apparent improvement in academic achievement, but the estimate may have a wide confidence interval compatible with substantially different effects.
Using the observed effect as definitive evidence can therefore be misleading.
Leon, Davis, and Kraemer caution against treating pilot studies as hypothesis-testing investigations when their primary purpose is feasibility.
Researchers should distinguish preliminary outcome information from evidence capable of establishing effectiveness.
What Does a Statistically Significant Pilot Result Mean?
A statistically significant result indicates that the observed data met the specified statistical criterion under the model and assumptions used.
It does not automatically establish that the intervention is effective, practically important, or likely to produce the same result in a definitive study.
In a small pilot, estimates may be unstable, and selective attention to significant findings can exaggerate apparent promise.
Researchers should consider effect estimates, confidence intervals, design limitations, and whether the comparison was planned.
A significant result may warrant further investigation, but it should not replace an appropriately designed definitive evaluation.
What Does a Nonsignificant Pilot Result Mean?
A statistically nonsignificant result does not establish that an intervention has no effect.
Small pilot studies often have limited statistical power to detect effects of plausible magnitude.
For example, a pilot may observe a modest difference between intervention and comparison groups but produce a wide confidence interval.
The appropriate conclusion may be that the pilot provides insufficiently precise evidence about effectiveness, not that the intervention failed.
Researchers should avoid treating failure to reject a null hypothesis as evidence of equivalence or absence of benefit.
Can a Pilot Establish Equivalence or Noninferiority?
Generally, not unless it was specifically designed and adequately powered for an appropriate equivalence or noninferiority analysis.
Equivalence requires evidence that a difference lies within a prespecified acceptable range, while noninferiority requires a suitable margin and inferential design.
A nonsignificant difference between groups does not establish equivalence.
This distinction is particularly important when researchers interpret a pilot as showing that two instructional approaches perform similarly.
Such a conclusion may be unsupported even when the observed group means are close.
Can a Pilot Demonstrate That an Intervention Is Safe?
A pilot can identify adverse events or implementation risks that occur during preliminary testing.
However, small samples and short follow-up periods are generally insufficient to rule out uncommon or delayed harms.
For example, a pilot of an educational technology intervention may reveal immediate usability or privacy concerns but fail to identify risks that emerge during sustained institutional use.
The absence of observed adverse events should not be interpreted as proof of safety.
Safety claims require evidence appropriate to the nature, frequency, and timing of the risks being evaluated.
Can Pilot Findings Be Generalized?
Generalizability depends on sampling, setting, design, and the nature of the claim.
Pilot participants may be recruited from a limited number of institutions or selected because they are especially willing to participate.
Such conditions may differ from those of the definitive study.
For example, teachers volunteering for a pilot of an AI-supported instructional tool may have greater interest or technical confidence than the wider teaching population.
The pilot can provide useful feasibility information under those conditions, but its findings should not automatically be generalized to all teachers.
Researchers should identify the contexts to which the feasibility evidence may reasonably apply.
How Should Pilot Percentages Be Reported?
Percentages should be accompanied by numerators, denominators, and relevant contextual information.
For example, reporting that 90% of participants completed follow-up is less informative than stating that 27 of 30 enrolled participants completed the assessment at the specified time point.
Where useful, confidence intervals can communicate uncertainty about the underlying proportion.
However, researchers should avoid interpreting a confidence interval as a guarantee that the definitive study's rate will fall within its bounds.
Precision should be evaluated in relation to the feasibility decision being made.
What Is the Difference Between Statistical Uncertainty and Feasibility Judgment?
Statistical uncertainty concerns the precision and variability of estimated quantities.
Feasibility judgment concerns whether the observed evidence supports proceeding with the proposed study, modifying it, or reconsidering the design.
The two are related but not identical.
For example, a pilot recruitment rate may be estimated imprecisely, yet operational evidence may clearly identify a remediable recruitment barrier.
Conversely, a seemingly favorable point estimate may be too uncertain to justify proceeding without further investigation.
Researchers should consider both numerical estimates and contextual information when making progression decisions.
How Do Progression Criteria Help?
Progression criteria specify the conditions under which a definitive study may proceed, require modification, or should not proceed without substantial redesign.
They may concern recruitment, retention, intervention fidelity, data completeness, or other feasibility outcomes.
For example, researchers might establish a minimum recruitment rate necessary to complete the definitive study within the available timeline.
However, thresholds should be justified by the specific research context rather than adopted arbitrarily.
Progression decisions should also consider uncertainty and whether observed problems can be addressed.
A traffic-light framework may be useful, but it should not replace methodological judgment.
What If a Pilot Meets Some Criteria but Not Others?
Mixed results are common because different procedures may have different levels of feasibility.
For example, recruitment may be adequate while intervention adherence is lower than anticipated.
The appropriate decision depends on the importance of each criterion and whether the problem can be remedied.
Researchers should not simply average feasibility indicators into one overall score unless such an approach has been justified.
A serious problem in a critical procedure may outweigh favorable findings elsewhere.
Progression decisions should consider the consequences of unresolved weaknesses for the definitive study.
Can Pilot Findings Support Modifying the Research Design?
Yes. Design modification is one of the principal purposes of preliminary feasibility research.
Pilot findings may justify revising recruitment procedures, reducing assessment burden, changing intervention training, or improving data management.
However, modifications should address identified problems rather than be selected merely to produce more favorable outcome estimates.
Researchers should document what changed, why it changed, and whether further testing is necessary.
Substantial changes may mean that the original pilot no longer provides sufficient evidence about the revised design.
Should Pilot Findings Determine the Definitive Sample Size?
Some pilot estimates can inform planning, but their uncertainty should be considered carefully.
Recruitment rates, retention estimates, and outcome variability may contribute to sample size or operational calculations.
However, using an unstable pilot treatment-effect estimate as the assumed effect for a definitive trial can lead to inappropriate sample size decisions.
Researchers should consider external evidence, substantively meaningful effects, and sensitivity analyses where appropriate.
The planning parameter should be selected according to the definitive study's objectives rather than whichever pilot estimate appears most favorable.
Can Pilot Data Be Included in the Definitive Study?
Sometimes. Internal pilots may be designed so that participants and data contribute to the definitive study, subject to prespecified procedures and methodological safeguards.
External pilot data are not automatically incorporated.
Whether inclusion is appropriate depends on compatibility of the design, intervention, measurements, eligibility criteria, consent, and statistical analysis.
Researchers should not combine datasets merely to increase sample size when substantial design differences undermine comparability.
How Should Unexpected Pilot Findings Be Interpreted?
Unexpected findings may reveal problems that were not included in the original feasibility objectives.
For example, a pilot may identify unanticipated teacher workload, technical difficulties, or participant confusion about assessment procedures.
Such findings can be important even when they were not prespecified.
Researchers should document them as exploratory or emergent observations and consider whether additional investigation is needed.
They should avoid retrospectively presenting unexpected findings as though they were the original primary objectives.
What Makes a Pilot Finding Actionable?
An actionable finding provides information that can reasonably influence a decision about the definitive study.
For example, evidence that participants frequently miss follow-up assessments because of scheduling conflicts may support revising assessment windows.
Evidence that recruitment takes longer than anticipated may support extending the recruitment period or adding sites.
By contrast, a small and imprecise difference in academic achievement may provide little reliable guidance about whether the intervention is effective.
The practical value of a finding depends on its relevance to the decision and the credibility of the evidence.
How Should Pilot Findings Be Reported?
Reports should begin with the pilot objectives and feasibility outcomes.
Researchers should provide appropriate numerators, denominators, estimates, uncertainty, and contextual information.
They should explain deviations from the protocol, unexpected difficulties, and modifications recommended for the definitive study.
Preliminary outcome estimates may be reported where justified, but they should be distinguished from definitive effectiveness evidence.
Reporting guidance such as the CONSORT extension for randomized pilot and feasibility trials can support transparency when applicable.
Watch Out
Do not interpret a statistically significant pilot outcome as proof of effectiveness or a nonsignificant result as proof of no effect. Pilot studies are generally designed to assess feasibility, and their preliminary effect estimates are often too imprecise for definitive conclusions.
