03 · What You Need to Know
What a Pilot Phase Should Establish Before a Definitive RCT
What Is a Pilot Randomized Controlled Trial?
A pilot randomized controlled trial is a smaller-scale implementation of a proposed definitive randomized trial, or an important part of that trial, undertaken primarily to investigate feasibility.
It may reproduce essential features of the planned study, including eligibility screening, recruitment, random allocation, intervention delivery, control procedures, and outcome assessment.
The central objective is not to determine conclusively whether the intervention works. It is to establish whether the larger trial can be conducted, whether proceeding is justified, and what modifications may be necessary.
The CONSORT extension for randomized pilot and feasibility trials emphasizes that pilot objectives should address uncertainties about the future definitive trial. Its reporting recommendations consequently differ from those for trials primarily evaluating effectiveness.
A pilot trial may collect preliminary outcome data, but such data should be interpreted according to the feasibility purpose and limited precision of the study.
What Is the Difference Between a Pilot Study and a Feasibility Study?
Feasibility research is the broader category. It investigates whether a proposed study can be undertaken and how its procedures might need to be developed or refined.
A pilot study is a particular form of feasibility research in which the intended definitive study, or a substantial component of it, is implemented on a smaller scale.
Feasibility study
Investigates whether and how a future study can be conducted; it need not reproduce the full trial or use randomization.
Pilot randomized trial
Tests the proposed randomized trial or important trial procedures on a smaller scale, with feasibility as the primary objective.
Definitive randomized trial
Is designed to evaluate the prespecified intervention effect with an appropriate level of precision and statistical power.
Terminology varies across disciplines and research programs. The important distinction concerns the study's objectives and procedures, rather than whether the researchers prefer the label pilot or feasibility.
When Is Recruitment Uncertainty a Reason to Pilot?
Recruitment difficulties can undermine an otherwise well-designed RCT. Investigators may overestimate the number of eligible participants, the proportion willing to enroll, or the rate at which recruitment sites can identify participants.
A pilot may be useful when these assumptions are uncertain and the definitive trial depends on achieving a demanding recruitment target.
Relevant feasibility measures include the number screened, proportion eligible, consent rate, recruitment rate per site, and time required to enroll participants.
For example, a university planning a trial involving 800 students may need evidence that participating departments can recruit the required sample within one academic year.
A pilot can test recruitment procedures and identify barriers. However, the resulting recruitment estimates should be interpreted with attention to the number of sites, their representativeness, and uncertainty around the observed rates.
When Should Intervention Delivery Be Piloted?
Complex interventions often require coordination among participants, intervention providers, and organizational systems. Even when the intervention is theoretically well developed, implementation may reveal unexpected difficulties.
A pilot may examine whether staff can deliver the intervention consistently, whether participants can access it, whether the intended dosage or duration is achievable, and whether implementation resources are sufficient.
For an AI-assisted teaching intervention, possible concerns include instructor training, technical access, integration with existing learning systems, and whether students actually engage with the assigned activities.
Intervention fidelity refers to the extent to which delivery corresponds to the intended protocol. Fidelity problems may weaken the treatment contrast or make results difficult to interpret.
A pilot can identify which components require refinement before a definitive trial, although it cannot guarantee that implementation will remain equally successful at a larger scale.
When Are Randomization and Allocation Procedures Worth Piloting?
Randomization methods can be statistically appropriate yet difficult to implement in particular settings.
For example, a cluster trial may encounter scheduling conflicts, resistance from institutions assigned to control, or contamination between groups. A trial involving several intervention arms may face challenges with allocation software, treatment availability, or participant eligibility.
A pilot can assess whether enrollment occurs before allocation where appropriate, whether assignment procedures are followed, and whether participants and providers understand the assigned conditions.
It may also reveal whether allocation concealment is operationally achievable and whether the proposed procedures create avoidable selection bias.
These issues should be addressed before the main trial begins because correcting them after extensive recruitment may be difficult.
When Should Outcome Measures Be Piloted?
An outcome measure may be established in the literature but difficult to administer in the intended population or study environment.
Researchers may need to determine whether participants understand questionnaire items, whether assessment sessions fit the intervention schedule, whether data can be extracted reliably, and whether outcome assessors can remain blinded when required.
A pilot can assess completion rates, missingness, administration time, data quality, and the practical suitability of proposed measurement procedures.
In educational technology studies, researchers might discover that learning analytics are recorded differently across course sections or that assessment instruments cannot be administered consistently within institutional timetables.
Such findings can justify revisions to data collection procedures before the definitive trial.
However, a small pilot should not automatically be treated as sufficient evidence of an instrument's validity or reliability. Psychometric evaluation may require separate methods and larger or more appropriate samples.
When Are Retention and Adherence Major Feasibility Concerns?
A definitive trial may require participants to complete repeated assessments, attend intervention sessions, or remain enrolled over a substantial follow-up period.
When the expected dropout rate is uncertain, a pilot can help assess retention and identify practical reasons for missing outcomes.
Adherence may also be important, particularly when intervention exposure depends on repeated participation.
Researchers should distinguish adherence to the assigned intervention from retention in outcome assessment. A participant may discontinue the intervention but still provide outcome data, and the appropriate analysis may require preserving that distinction.
Observed pilot attrition rates are uncertain estimates, especially in small samples. They can inform planning, but they should not be assumed to predict definitive-trial attrition exactly.
What Is the Difference Between an External and an Internal Pilot?
An external pilot is conducted separately before the definitive trial. Its participants are generally not included in the main trial's analysis.
An internal pilot is incorporated into the definitive trial, with an initial phase used to assess specified feasibility criteria. Participants from that phase may remain in the final analysis when the design, eligibility, interventions, and prespecified procedures permit.
| Feature |
External pilot |
Internal pilot |
| Relationship to main trial |
Separate preliminary study |
Embedded initial phase |
| Participant data |
Generally excluded from definitive analysis |
May contribute to final analysis under prespecified conditions |
| Design modification |
Can accommodate substantial revisions before the main trial |
Usually requires more constrained and carefully planned modifications |
| Potential advantage |
Greater flexibility to refine procedures |
May avoid discarding usable participant data |
| Important concern |
Additional time and resources |
Changes must preserve validity of the definitive trial |
An internal pilot should not be confused with an unplanned interim analysis of treatment effectiveness. Its feasibility assessments, progression rules, and permissible modifications should be specified prospectively.
How Do You Set Progression Criteria?
Progression criteria define how pilot findings will inform the decision to proceed to a definitive trial, modify the proposed design, or stop development.
They should be tied to the most consequential feasibility uncertainties rather than selected merely because they are easy to measure.
For example, researchers might establish minimum acceptable recruitment rates, outcome completion levels, or intervention delivery standards.
A commonly used approach classifies feasibility results into three categories: proceed, proceed with modifications, or do not proceed without substantial reconsideration.
These thresholds should be justified using the requirements of the proposed definitive trial, available resources, and relevant prior evidence. They are not universal standards.
Progression criteria should also be interpreted with uncertainty in mind. A small pilot's observed rate may be an imprecise estimate of the underlying feasibility parameter.
The final decision may appropriately incorporate quantitative results, qualitative findings, participant safety, and practical considerations.
How Large Should a Pilot RCT Be?
Pilot sample size should be determined by the feasibility objectives and the precision needed to inform subsequent decisions.
There is no universally appropriate number of participants per arm.
A pilot focused on recruitment feasibility may require enough participants and recruitment sites to estimate enrollment rates meaningfully. A pilot examining intervention delivery may need sufficient exposure across different providers or settings to identify operational difficulties.
For a single proportion estimated from independent participants, a simple approximate standard error is:
The calculation illustrates uncertainty around a feasibility proportion. For small samples or proportions near zero or one, Wilson or exact binomial intervals may be more appropriate. Clustered recruitment or repeated observations require methods that account for dependence.
Researchers should not choose a pilot sample solely because it is a convenient fraction of the definitive trial sample.
Recent methodological work on pilot sample-size planning emphasizes matching calculations to feasibility objectives, including the precision of parameter estimates and the evaluation of progression criteria.
Should a Pilot Trial Test Whether the Intervention Is Effective?
Ordinarily, a pilot RCT should not use formal hypothesis testing of effectiveness as its principal objective.
Pilot studies are generally too small to provide precise treatment-effect estimates. A nonsignificant result may reflect inadequate power, while a statistically significant result may be unstable or exaggerated.
Preliminary outcome estimates can be reported descriptively with uncertainty intervals where useful, but they should not be presented as definitive evidence of effectiveness.
Likewise, using an imprecise pilot treatment-effect estimate as the sole basis for a definitive trial's sample-size calculation can be problematic. The target effect should be scientifically meaningful and informed by relevant evidence.
Some early-phase studies are specifically designed to investigate preliminary efficacy or other scientific questions. Their objectives may differ from those of a feasibility pilot and should be described accordingly.
When Is a Separate Pilot Phase Unnecessary?
A separate pilot may be unnecessary when recruitment pathways are established, intervention delivery has been demonstrated in comparable settings, outcome measures are suitable, and the proposed trial procedures are already well understood.
For example, researchers repeating an established trial procedure in a similar population may have sufficient feasibility evidence from prior studies and operational records.
Some uncertainties can also be resolved through simpler preparatory activities, such as usability testing, stakeholder consultation, staff training, or a limited rehearsal of data collection procedures.
A pilot should be chosen because it can resolve a meaningful uncertainty, not because the research team believes every RCT must begin with a smaller RCT.
If the main uncertainty concerns whether the intervention has a scientifically worthwhile effect rather than whether the trial can be implemented, a feasibility pilot may not be the appropriate next study.
What Should Happen After the Pilot?
The research team should compare observed feasibility results with the prespecified progression criteria and examine any important unintended consequences.
Possible decisions include proceeding without major changes, revising recruitment or intervention procedures, undertaking additional feasibility work, or deciding that the definitive trial is not currently justified.
A decision not to proceed does not necessarily mean the pilot failed. Identifying an unworkable design before committing substantial resources can be a valuable research outcome.
The pilot report should explain what was learned, what modifications are proposed, and how the findings affect the feasibility of the definitive trial.
