03 · What You Need to Know
Building a Defensible Assessment of Intervention Delivery
What Does It Mean to Assess Intervention Delivery?
Intervention delivery assessment examines whether providers implemented the intended intervention components and procedures.
It is a central aspect of intervention fidelity, although fidelity frameworks may also include provider training, participant receipt, enactment, and other dimensions.
For example, a research writing intervention may require diagnostic assessment, individualized feedback, guided revision, and reflection activities.
Assessing delivery involves establishing whether these components were provided, whether the intended sequence or decision rules were followed, and whether delivery met relevant quality criteria.
The assessment should be based on a sufficiently detailed intervention description. Without such a description, researchers cannot reliably determine whether observed delivery corresponds to the intended treatment.
Begin With an Intervention Specification
The first requirement is a clear account of what providers are supposed to deliver.
The specification should identify essential activities, their intended frequency or duration, provider responsibilities, relevant materials, and permissible adaptations.
For example, a digital feedback program might require each student to receive diagnostic feedback after a formative assessment, followed by an opportunity to revise their work.
These requirements can be translated into observable indicators.
Researchers should distinguish essential components from optional activities and ordinary practices permitted in both treatment conditions.
The TIDieR checklist provides guidance for describing interventions in enough detail to support replication and assessment of planned and actual delivery.
What Should Be Measured During Delivery?
Delivery fidelity commonly involves content adherence, dose delivered, and quality of implementation.
Depending on the intervention, sequence, timing, differentiation, and adherence to adaptation rules may also matter.
| Delivery dimension |
Assessment question |
Possible evidence |
| Content adherence |
Were required components delivered? |
Component checklist |
| Dose delivered |
How much intervention was provided? |
Session counts and duration records |
| Delivery quality |
Were procedures implemented competently? |
Structured observer ratings |
| Sequence and timing |
Were components delivered in the intended order or window? |
Dated delivery records |
| Permitted adaptation |
Did modifications follow the intervention rules? |
Adaptation documentation |
| Intervention differentiation |
Were condition-specific practices maintained? |
Observations across treatment conditions |
Researchers should select dimensions according to the intervention's expected mechanism and the information needed to interpret the study.
How Do You Develop an Intervention Delivery Checklist?
A delivery checklist translates intervention requirements into observable items.
Each item should describe an action that can be assessed consistently rather than a vague impression of implementation quality.
For example, an item such as "Instructor delivered appropriate feedback" may be too broad unless appropriate feedback has been operationally defined.
A more specific item might state that the instructor identified the student's principal error, explained the relevant criterion, and provided an opportunity for revision.
Checklists may use binary responses, frequency counts, or graded ratings depending on the nature of the component.
Researchers should pilot the instrument and revise ambiguous items before using it for formal assessment.
How Do You Calculate a Content Adherence Score?
When required components can be classified as delivered or not delivered, researchers may calculate the proportion completed.
The calculation is straightforward, but interpretation requires caution.
It assumes the components are appropriately counted and, when combined into a simple percentage, implicitly gives them equal numerical weight.
Omitting one essential component may be more consequential than omitting several peripheral activities.
Researchers should therefore report important component-level findings rather than rely exclusively on an overall percentage.
Should Every Component Receive Equal Weight?
Not necessarily. Some intervention components may be more central to the intended mechanism than others.
For example, an intervention designed around individualized diagnostic feedback may depend more strongly on that feedback than on an introductory orientation activity.
Researchers may consider separate reporting of core components or theoretically justified weighting.
However, weights should not be invented after examining which components are associated with favorable outcomes.
Any weighted score requires a clear rationale, defined scoring procedures, and appropriate evaluation of its measurement properties.
When evidence supporting weights is limited, reporting component-specific adherence may be more transparent than constructing a single composite score.
How Can Direct Observation Improve Fidelity Assessment?
Direct observation allows trained assessors to examine intervention delivery as it occurs or through recorded sessions.
Observers can determine whether activities were delivered and may also evaluate the quality of provider behavior.
For example, an observer may assess whether an instructor gives diagnostic feedback, invites student revision, and follows the intervention's intended interaction sequence.
However, observation requires resources and may influence provider behavior.
Researchers should consider how sessions are selected, whether observers know treatment assignments, and whether observations adequately represent implementation across providers and time.
Observation is valuable, but a small number of highly prepared sessions may not represent routine delivery.
How Should Sessions Be Sampled for Observation?
Observing every session may be impractical, especially in multi-site or long-duration studies.
Researchers can select sessions using a sampling strategy that reflects relevant implementation variation.
For example, the plan might include observations from early, middle, and later implementation periods and from different providers.
Random sampling of eligible sessions may reduce selective observation, while stratified sampling may ensure representation of important delivery contexts.
The appropriate strategy depends on the number of sessions, expected variation, and available resources.
Researchers should document which sessions were observed and avoid generalizing beyond the evidence collected.
How Do You Assess Delivery Quality?
Quality assessment concerns how competently providers perform the intervention procedures.
It differs from simply recording whether activities occurred.
For example, an instructor may provide feedback at every required session but give vague comments that do not identify errors or support revision.
A quality rubric might evaluate the specificity, relevance, and procedural completeness of the feedback.
Rating criteria should describe observable behaviors and avoid relying entirely on global impressions.
Researchers should also consider whether the rating instrument has adequate content validity and whether assessors can apply it consistently.
The related question of measuring intervention provider competence may require more detailed assessment of professional performance.
How Can You Assess Agreement Between Fidelity Raters?
When multiple observers score the same delivery behaviors, their ratings should be sufficiently consistent for the intended use.
Researchers may evaluate inter-rater agreement using methods appropriate to the scale and data structure.
For categorical ratings, agreement statistics such as Cohen's kappa may be relevant in suitable circumstances. For continuous or ordinal ratings, other reliability approaches may be more appropriate.
No single statistic is universally suitable.
Researchers should train raters, establish clear scoring rules, and investigate disagreements that reveal ambiguous definitions.
High agreement alone does not establish that the instrument measures the intended fidelity construct correctly.
Can Provider Self-Reports Be Used?
Provider self-reports can offer practical information about what was delivered, especially when direct observation is limited.
For example, instructors may complete session checklists recording activities, duration, and adaptations.
However, self-reports can be affected by recall error, misunderstanding, or pressure to demonstrate compliance.
Providers may also fail to recognize that their delivery differed from the intended procedure.
Researchers should therefore consider corroborating important self-reported information through observations, records, or other sources where feasible.
The usefulness of self-report depends on the specificity of the items and the consequences of measurement error.
What Can Digital Intervention Logs Establish?
Digital platforms can automatically record events such as content release, session duration, feedback generation, and activity completion.
These records may provide detailed information about delivery timing and exposure.
For example, a platform log might confirm that individualized feedback was generated and made available to a student.
However, this does not necessarily establish that the feedback was appropriate, understood, or acted upon.
Researchers should distinguish system-generated events from participant receipt and meaningful engagement.
Digital logs can strengthen delivery assessment but do not replace every other fidelity measure.
How Is Dose Delivered Different From Dose Received?
Dose delivered concerns the amount of intervention provided by the research team or intervention system.
Dose received concerns the amount participants actually encounter or engage with, depending on the framework and operational definition.
For example, an instructor may deliver eight workshops while a participant attends only five.
The delivery record and attendance record describe different aspects of implementation.
Likewise, a digital platform may release all required activities even when participants complete only a portion.
Researchers should measure intervention dose in a way that distinguishes provision from receipt.
How Should Adaptations Be Recorded?
Intervention providers may adapt materials, examples, timing, or delivery procedures to suit participants and local conditions.
Some adaptations may be consistent with the intervention's intended flexibility. Others may change essential components.
Researchers should document what was modified, why, when, by whom, and whether the modification was planned or reactive.
The FRAME framework provides a structured approach to reporting intervention adaptations and modifications.
Researchers should avoid automatically scoring every adaptation as a fidelity failure.
Instead, the assessment should determine whether the modification preserved the intervention's essential functions and followed permitted adaptation rules.
Should Fidelity Be Assessed in the Control Group?
Sometimes. The comparison condition may also require implementation assessment.
For example, a control group described as receiving conventional instruction might actually receive elements of the experimental program through staff behavior.
Alternatively, usual care may differ substantially from what the protocol anticipated.
Assessing the comparison condition can help establish intervention differentiation and identify contamination.
However, the measurement strategy should reflect the comparison condition's intended structure.
A flexible usual-care condition should not automatically be judged against the same fixed delivery checklist used for the experimental intervention.
How Can Fidelity Monitoring Be Used During Implementation?
Monitoring can identify delivery problems while the study is underway.
For example, early observations may reveal that providers consistently omit a required activity.
Researchers may then provide corrective training or clarification if this is consistent with the protocol.
However, feedback and corrective support may change implementation over time.
The monitoring plan should therefore specify when feedback is provided, who receives it, and how corrective actions are documented.
Researchers should distinguish routine implementation support from unplanned modifications introduced after examining outcome results.
Should Researchers Establish a Fidelity Threshold?
Operational thresholds can be useful when they are justified by the intervention design.
For example, a study may specify that certain essential procedures must always be completed or that repeated omissions trigger additional training.
However, a single percentage does not establish adequate fidelity across all interventions.
A session with 90% checklist completion may still omit the component most central to the intervention's intended mechanism.
Researchers should therefore report important component-level deviations and distinguish monitoring thresholds from validated evidence of sufficient implementation.
Can Fidelity Scores Be Related to Participant Outcomes?
Researchers may investigate whether variation in fidelity is associated with variation in outcomes.
Such analyses can provide useful exploratory information about implementation.
However, fidelity is generally observed after treatment assignment and may be influenced by provider characteristics, participant needs, and contextual factors.
Participants receiving higher-fidelity delivery may differ from those receiving lower-fidelity delivery in ways related to outcomes.
Consequently, an association between fidelity and outcomes does not automatically establish that increasing fidelity would cause improvement.
Causal interpretation requires an appropriate design or additional assumptions.
Should Participants Be Excluded When Fidelity Is Low?
Not automatically. Excluding participants according to post-randomization implementation may introduce selection bias.
For example, providers may deliver fewer intervention activities to participants who experience difficulties or miss sessions.
Removing these participants could create groups that are no longer comparable under the original randomization.
The primary analysis should follow the prespecified estimand and appropriate analytical strategy.
Additional analyses examining fidelity variation may be informative, but they should be distinguished from the primary randomized comparison.
How Should Delivery Fidelity Be Reported?
Reports should identify the intervention components assessed, measurement instruments, sampling procedures, observers, scoring rules, and relevant implementation periods.
Researchers should describe actual delivery rather than merely state that fidelity was monitored.
Where feasible, component-level results should accompany overall summaries.
Reports should also identify permitted adaptations, consequential deviations, and limitations in the fidelity evidence.
The TIDieR checklist and Medical Research Council process evaluation guidance provide useful foundations for reporting intervention implementation.
Fidelity findings should support interpretation of what was tested without being presented as proof that implementation differences caused the observed outcomes.
