03 · What You Need to Know
How to Evaluate Intervention Delivery Before the Definitive Study
What Does Pilot Testing Intervention Delivery Mean?
Intervention delivery piloting involves implementing a proposed intervention, or selected components of it, on a limited scale to examine whether the intended procedures can be carried out.
The pilot may assess staff training, scheduling, resource availability, intervention fidelity, participant engagement, technical functionality, and the practical demands of implementation.
For example, a university may plan to introduce an AI-supported formative feedback system into several online courses.
A pilot could examine whether teachers can configure the system, use it during planned learning activities, review generated feedback, and maintain appropriate instructional oversight.
The purpose is to understand whether the intervention can operate as intended within the proposed research setting.
How Is Delivery Feasibility Different From Intervention Effectiveness?
Delivery feasibility concerns whether the intervention can be implemented according to its intended procedures.
Effectiveness concerns whether the intervention produces its intended outcomes.
Intervention delivery feasibility
Whether the intervention can be implemented with suitable personnel, resources, timing, fidelity, and participant involvement.
Intervention effectiveness
Whether the intervention produces the intended outcomes under the conditions and comparisons established by the research design.
For example, teachers may successfully deliver every planned AI-supported feedback activity without the intervention improving student achievement.
Conversely, preliminary outcome improvements may occur even when delivery is inconsistent, making it difficult to understand which intervention components were actually received.
Both questions matter, but they require different evidence.
When Is Delivery Piloting Particularly Important?
Delivery piloting may be especially useful when the intervention is newly developed, involves multiple components, requires unfamiliar skills, depends on technical infrastructure, or must be implemented across several sites.
It may also be valuable when an established intervention is transferred to a substantially different population or context.
For example, an instructional strategy developed for small face-to-face classes may require adaptation before being implemented in large asynchronous online courses.
Existing evidence of effectiveness does not necessarily establish feasibility in the new setting.
Researchers should examine the degree of uncertainty rather than assume that every intervention requires a separate pilot.
What Intervention Delivery Components Should Be Tested?
| Component |
What to examine |
Potential problem |
| Staff preparation |
Whether implementers receive and understand necessary training |
Inconsistent interpretation of procedures |
| Intervention fidelity |
Whether intended components are delivered as specified |
Omitted or substantially modified activities |
| Intervention dose |
How much intervention is delivered and received |
Insufficient exposure to planned activities |
| Participant adherence |
Whether participants engage with required activities |
Incomplete participation |
| Scheduling |
Whether activities fit within the research and institutional timetable |
Conflicts with ordinary operations |
| Resources |
Whether necessary personnel, materials, and infrastructure are available |
Implementation delays or interruptions |
| Acceptability |
How participants and implementers experience the intervention |
Excessive burden or perceived inappropriateness |
| Technical functionality |
Whether required systems operate under realistic conditions |
Software errors or access difficulties |
| Monitoring |
Whether delivery deviations and implementation problems can be recorded |
Incomplete evidence about what occurred |
The appropriate selection depends on the intervention and the uncertainty the pilot is intended to resolve.
What Is Intervention Fidelity?
Intervention fidelity concerns the extent to which an intervention is delivered according to its intended design.
It may involve adherence to prescribed content, quality of delivery, differentiation from comparison conditions, and other relevant dimensions.
For example, an educational intervention may require teachers to provide formative feedback after specified learning activities.
A pilot could examine whether feedback is delivered at the intended points and whether its content reflects the intervention protocol.
However, fidelity should not be reduced automatically to completing a checklist.
The quality of implementation may matter, and some interventions intentionally permit adaptation to participants or contexts.
Does High Fidelity Mean the Intervention Is Effective?
No. High fidelity indicates that the intervention was delivered in a manner consistent with the specified procedures.
It does not establish that the intervention produces the intended outcomes.
For example, a teacher may implement every planned AI feedback activity correctly, yet students may experience little measurable improvement.
The pilot can support a conclusion about delivery under the observed conditions, but effectiveness requires appropriate outcome evaluation.
Researchers should avoid treating successful implementation as a substitute for evidence of educational benefit.
How Is Fidelity Different From Adherence?
Fidelity and adherence are related concepts, but their meanings depend on the implementation framework and intervention.
Fidelity commonly concerns whether the intervention is delivered as intended. Adherence may refer to whether implementers follow the protocol or whether participants complete prescribed activities.
For example, a teacher may deliver all planned learning activities while some students complete only a portion of them.
The intervention may therefore have high delivery fidelity but lower participant adherence.
Researchers should define these terms operationally rather than assuming that all studies use them identically.
What Is Intervention Dose?
Intervention dose concerns the amount of intervention delivered or received.
It may be represented by the number of sessions, duration of exposure, completion of activities, or another measure appropriate to the intervention.
For example, a six-week instructional intervention may require one structured feedback activity each week.
A pilot can determine whether the planned activities can be delivered and whether students have opportunities to complete them.
However, more exposure is not necessarily better. The intended dose should be justified by the intervention design and relevant evidence.
What Is the Difference Between Dose Delivered and Dose Received?
Dose delivered concerns the amount of intervention provided by implementers.
Dose received concerns the amount participants actually experience or engage with.
For example, a teacher may provide six planned feedback activities, but a student may access only four.
These observations describe different implementation processes.
Researchers should avoid assuming that providing an activity guarantees meaningful participant engagement.
The appropriate measures depend on whether exposure, participation, comprehension, or another dimension is relevant to the intervention.
Should Participant Acceptability Be Evaluated?
Acceptability may be important when the intervention introduces unfamiliar practices, additional workload, sensitive activities, or changes to established routines.
Researchers may examine whether participants consider the intervention appropriate, manageable, and worthwhile.
For example, teachers may value AI-supported feedback but find the required verification process difficult to sustain alongside other responsibilities.
Such findings can inform implementation modifications.
However, favorable acceptability does not establish effectiveness, and unfavorable initial reactions do not necessarily mean that the intervention cannot be implemented.
Acceptability should be interpreted in relation to the participants, setting, and stage of implementation.
How Can Acceptability Be Assessed?
Acceptability may be examined through interviews, questionnaires, observations, or other appropriate methods.
Researchers might ask participants about perceived burden, coherence, opportunity costs, or concerns associated with the intervention.
Sekhon, Cartwright, and Francis developed a theoretical framework of acceptability that identifies several dimensions relevant to healthcare interventions. The framework may inform thinking in other settings, although its suitability should be considered rather than assumed.
Researchers should select assessment procedures that correspond to the intervention and intended implementation context.
For example, teacher interviews may reveal why an intervention is difficult to deliver even when completion records suggest reasonable adherence.
Should Intervention Training Be Piloted?
Yes, when successful delivery depends on implementers acquiring unfamiliar knowledge or skills.
A pilot can examine whether training materials are understandable, whether the allocated training time is sufficient, and whether implementers can perform required activities.
For example, teachers using an AI-supported feedback system may need to understand how to review generated responses, identify inaccuracies, and intervene when feedback is unsuitable.
Attendance at training does not automatically establish competence.
Researchers may need demonstrations, observations, or other suitable assessments of implementation readiness.
How Should Technical Requirements Be Tested?
Technical testing may be important when the intervention depends on software, devices, internet connectivity, data exchange, or institutional systems.
Researchers should examine whether the intervention operates under realistic conditions rather than only in controlled demonstrations.
For example, an online feedback system may function with a small test dataset but experience delays when several classes use it simultaneously.
A pilot can identify access difficulties, configuration problems, interruptions, and support requirements.
However, technical functionality is not equivalent to educational suitability or participant acceptability.
What If the Intervention Is Delivered Differently Across Sites?
Multisite studies may encounter variation in staffing, institutional schedules, infrastructure, and local practices.
Some variation may represent legitimate contextual adaptation, while other differences may undermine the intended intervention.
Researchers should identify which components are essential and which can be adapted without changing the intervention's underlying rationale.
A pilot can examine whether these distinctions are practical.
For example, universities may schedule feedback activities differently while preserving the essential instructional function.
However, allowing unrestricted variation may make it difficult to determine what intervention was actually evaluated.
Should You Identify Core Components Before Piloting?
Generally, yes. Core components are elements considered essential to the intervention's intended function.
Researchers should distinguish these from procedures that may be adapted to local circumstances.
For example, an intervention may require timely, individualized formative feedback but allow flexibility in how teachers schedule feedback sessions.
The distinction should be grounded in theory, intervention development, and available evidence.
A pilot can reveal whether the proposed core components are feasible and whether permitted adaptations preserve their intended function.
However, the pilot may also expose weaknesses in the original assumptions about what is essential.
Can a Pilot Identify Implementation Barriers?
Yes. A pilot may reveal barriers involving staffing, workload, technology, participant expectations, institutional procedures, or competing priorities.
For example, teachers may be willing to deliver an intervention but unable to complete required activities during examination periods.
Researchers should investigate the nature of these barriers rather than merely record that implementation was incomplete.
Some problems may be resolved through scheduling or training. Others may indicate that the intervention is incompatible with the intended setting.
The value of the pilot lies partly in distinguishing remediable problems from more fundamental feasibility limitations.
How Should Implementation Barriers Be Documented?
Researchers may use structured implementation logs, observations, staff feedback, participant interviews, or other appropriate records.
Documentation should distinguish what was planned, what occurred, why deviations arose, and what consequences they may have had.
For example, a missed intervention session could reflect technical failure, participant absence, or an intentional adaptation by the teacher.
These explanations have different implications for the definitive study.
Researchers should avoid treating all deviations as equivalent without examining their causes.
Should You Test Intervention Delivery and Outcome Assessment Together?
Sometimes. The relationship between delivery and measurement may create operational problems.
For example, outcome assessments may be scheduled immediately after intervention activities, but participants may not have sufficient time to complete both.
Alternatively, implementers may inadvertently influence outcome assessment when they know participants' intervention assignments.
A pilot can examine whether delivery and measurement procedures can operate together without compromising the study design.
However, outcome assessment has its own methodological requirements, including measurement quality and appropriate handling of missing data.
See Should You Pilot Test Outcome Assessment Procedures?.
Can Intervention Delivery Be Tested Without Randomization?
Yes. A single-arm feasibility study or focused implementation pilot may be appropriate when the primary uncertainty concerns whether the intervention can be delivered.
For example, researchers may first test whether teachers can implement a new instructional workflow before examining whether it can be evaluated through a randomized comparison.
However, a nonrandomized delivery pilot cannot establish the feasibility of randomization procedures that were never tested.
The preliminary design should correspond to the uncertainties requiring investigation.
Should You Pilot the Intervention With the Intended Population?
Generally, pilot participants and settings should be sufficiently similar to those of the definitive study to provide relevant implementation evidence.
For example, an intervention intended for first-year online students may operate differently when tested only with experienced graduate students.
However, exact equivalence is not always possible, particularly during early development.
Researchers should identify which contextual differences may limit transferability of the findings.
Successful implementation in one institution does not guarantee comparable delivery across all intended sites.
How Large Should an Intervention Delivery Pilot Be?
There is no universal sample size.
The appropriate scope depends on the intervention's complexity, the feasibility outcomes, variation across implementers or settings, and the precision required for relevant estimates.
A small pilot may identify obvious training or workflow problems but may not capture variation across institutions or uncommon technical failures.
For cluster-delivered interventions, the number and diversity of clusters may be particularly consequential.
Researchers should justify the sample size according to the implementation questions rather than an arbitrary fraction of the definitive sample.
What Progression Criteria Should Be Used?
Progression criteria may specify conditions under which the intervention can proceed to definitive evaluation, requires modification, or should not proceed without substantial redesign.
Relevant criteria may concern fidelity, adherence, staff training, technical reliability, participant acceptability, or resource requirements.
For example, a study may establish a contextually justified minimum proportion of planned intervention sessions that must be delivered.
However, a single percentage may conceal serious problems with essential components.
Progression decisions should consider the nature of deviations, uncertainty, and whether problems are remediable.
Should You Modify the Intervention After the Pilot?
Yes, when the findings justify modifications.
Researchers may revise training, scheduling, technical support, participant instructions, or other implementation procedures.
However, substantial changes to core components may alter the intervention itself.
If the revised intervention differs materially from what was piloted, additional feasibility testing may be necessary.
Researchers should document what changed and why, particularly when the definitive study will evaluate a modified version.
Can Pilot Intervention Data Be Included in the Definitive Study?
Sometimes, particularly in appropriately planned internal pilots.
However, inclusion depends on the research design, consent arrangements, compatibility of intervention versions, and statistical considerations.
Data collected under substantially different intervention conditions may not be suitable for pooling with definitive-study data.
Researchers should not combine observations merely to increase sample size.
The decision should be established prospectively where possible and justified according to the intended analysis.
How Should Intervention Delivery Pilot Findings Be Interpreted?
Findings should be interpreted according to the feasibility objectives and observed implementation conditions.
For example, a pilot may show that most planned sessions were delivered, but staff required more time than anticipated.
This may support proceeding with modifications to training or workload allocation.
It does not establish that the intervention is effective or that implementation will remain equally successful at larger scale.
The distinction between actionable feasibility evidence and unsupported effectiveness claims is examined in How Do You Distinguish Useful Pilot Study Findings From Conclusions a Pilot Cannot Support?.
How Should Intervention Delivery Be Reported?
The report should describe the intervention components, intended delivery procedures, implementers, setting, feasibility outcomes, and observed deviations.
Researchers should explain how fidelity, adherence, dose, and acceptability were defined and assessed.
Where applicable, the Template for Intervention Description and Replication (TIDieR) checklist can support sufficiently detailed intervention reporting.
The Medical Research Council framework for complex interventions also provides relevant guidance concerning development, feasibility, evaluation, and implementation.
However, frameworks developed primarily in health research should be adapted thoughtfully when applied to educational interventions.
Watch Out
Do not interpret successful intervention delivery as proof of effectiveness. A pilot may demonstrate that the intervention can be implemented, but determining whether it produces the intended outcomes requires an appropriate evaluation design and sufficiently informative evidence.
