03 · What You Need to Know
Recognizing When Intervention Flexibility Threatens Treatment Fidelity and Interpretation
What Is Intervention Flexibility?
Intervention flexibility refers to the degree to which an intervention's content, activities, delivery methods, timing, dose, or other features may vary during implementation.
Some flexibility is deliberately designed into the intervention. An adaptive learning system, for example, may select different activities according to students' prior knowledge and progress.
Other variation may arise because providers interpret the protocol differently or modify activities in response to local circumstances.
These forms of variation should not be treated as equivalent.
Flexibility is methodologically defensible when it is part of a sufficiently specified treatment strategy. It becomes more problematic when providers effectively determine what the intervention means without common boundaries.
What Makes Intervention Flexibility Excessive?
There is no universal numerical threshold at which flexibility becomes excessive.
The concern arises when variation prevents researchers from establishing the intervention's defining features or understanding what participants were assigned to receive.
For example, an intervention intended to teach critical evaluation of AI-generated research suggestions may lose its defining function if some providers replace that activity with unrestricted use of generative AI tools.
The problem is not merely that the activities differ. It is that the modification may remove the process through which the intervention was expected to produce its effect.
Researchers should therefore evaluate flexibility according to its consequences for intervention identity, causal theory, fidelity, and the intended treatment comparison.
Why Does Intervention Identity Matter?
An intervention must be described sufficiently for researchers and readers to understand the treatment strategy being evaluated.
If the intervention is defined only as enhanced academic support, different institutions may implement entirely different services under that label.
A randomized trial could still estimate the effect of assignment to such a broad program as implemented, provided the assignment and analytical procedures are appropriate.
However, the estimate may be difficult to interpret beyond the specific combination of activities, providers, and settings included in the study.
Researchers should distinguish identifying an average effect of a heterogeneous intervention package from identifying the effects of particular components.
The first may be possible even when the second remains uncertain.
How Is Excessive Flexibility Different From Appropriate Adaptation?
Appropriate adaptation preserves the intervention's intended functions while modifying aspects of delivery according to defined needs or circumstances.
Excessive flexibility may remove or replace essential functions, permit incompatible intervention versions, or leave important modifications undocumented.
Appropriate intervention adaptation
Modification within defined boundaries that preserves the intended treatment strategy and its essential functions.
Problematic intervention flexibility
Variation that changes defining components, lacks clear decision rules, or prevents researchers from identifying what treatment was implemented.
These classifications require a defensible intervention specification. A change cannot be reliably judged against fidelity requirements that were never established.
What Is Intervention Drift?
Intervention drift describes changes in implementation over time that move delivery away from the originally intended procedures or functions.
Drift may occur gradually rather than through one obvious protocol deviation.
For example, an instructor may initially conduct every required reflection activity but gradually shorten those activities as other course demands increase.
Another provider may replace guided revision with general advice because the replacement is easier to deliver.
Not every change over time constitutes problematic drift. Some interventions are designed to evolve or adapt according to specified rules.
The concern is whether changes move implementation outside the intended strategy without appropriate justification or documentation.
How Can Flexibility Affect the Treatment Contrast?
In comparative research, the treatment contrast concerns the differences between the intervention strategies being evaluated.
If providers substantially modify the experimental intervention, participants may receive different versions of the treatment.
Some versions may resemble the comparison condition more closely than others.
For example, a structured feedback intervention may become indistinguishable from ordinary instructor consultations if providers remove its diagnostic and revision components.
Such variation can reduce intervention differentiation and complicate interpretation of the observed group difference.
However, the direction of the effect is not predictable. Different versions may produce stronger, weaker, or qualitatively different outcomes.
Does Intervention Heterogeneity Automatically Create Bias?
No. Heterogeneous treatment delivery is not automatically a source of bias for every estimand.
A pragmatic trial may intentionally evaluate a flexible intervention package delivered under ordinary institutional conditions.
If the strategy explicitly permits provider discretion, variation may be part of the treatment being evaluated.
The randomized comparison may still estimate the effect of assignment to that strategy under the studied implementation conditions.
However, broad variation may complicate explanations of why the intervention worked and whether findings apply elsewhere.
Researchers should distinguish threats to causal identification from limitations in treatment definition, mechanistic interpretation, and transportability.
How Can Researchers Identify Core Components That Must Remain Intact?
The intervention's causal theory should identify components believed to activate its intended mechanisms.
Researchers can draw on prior studies, intervention development work, practitioner knowledge, and feasibility evidence to specify candidate core components.
For example, a research writing intervention may require students to identify weaknesses in arguments, evaluate AI-generated suggestions, and justify revisions.
Different examples or delivery formats may be permissible if those functions remain intact.
However, identifying a component as core does not prove that it is causally necessary.
Researchers should distinguish theoretically justified intervention requirements from empirically demonstrated active ingredients.
Can Two Different Activities Serve the Same Intervention Function?
Potentially. Written feedback and a structured oral consultation might both be intended to help participants recognize errors and revise their work.
However, similar intended functions do not guarantee equivalent effects.
Delivery format may influence participant understanding, engagement, accessibility, or the amount of feedback received.
Researchers should therefore specify the criteria under which alternative activities are considered acceptable.
When the equivalence of delivery formats is uncertain and consequential, further empirical evaluation may be appropriate.
Functional flexibility should be justified rather than used as a general explanation for unlimited modification.
What Types of Flexibility Are Most Likely to Threaten Interpretability?
| Type of variation |
Potential concern |
Possible safeguard |
| Removal of essential content |
The intended causal function may no longer be delivered. |
Define mandatory components and monitor their presence. |
| Unrestricted provider substitution |
Providers may replace the intervention with different activities. |
Specify permitted alternatives and review requirements. |
| Uncontrolled dose variation |
Participants may receive substantially different treatment exposure. |
Define dose ranges or adaptive decision rules. |
| Inconsistent intervention sequence |
Activities may be delivered without necessary prerequisites. |
Identify sequence dependencies and permitted timing changes. |
| Undocumented local modifications |
Researchers may not know what was delivered at each site. |
Maintain proportionate adaptation and delivery records. |
| Changing provider competence |
Delivery quality may differ substantially across providers. |
Establish training and observable competence criteria. |
These are potential risks rather than automatic evidence of invalidity. Their importance depends on the intervention and target estimand.
How Does Dose Variation Complicate Interpretation?
When providers vary the amount of intervention delivered, participants may receive substantially different exposures.
For example, one instructor may provide weekly guided revision while another offers only occasional feedback.
If the intervention is intended to use a fixed dose, these differences may represent implementation deviations.
If dose is deliberately adaptive, variation may be consistent with the treatment strategy.
Researchers should distinguish dose delivered from dose received and identify the rules governing intended variation.
Additional considerations are discussed in measuring intervention dose.
How Does Provider Competence Influence Flexibility?
Provider discretion depends partly on the ability to recognize which intervention functions must be preserved.
A provider may adapt examples appropriately while maintaining the intended instructional process. Another may unintentionally remove essential activities while attempting to simplify delivery.
Experience may support adaptation decisions, but credentials alone do not establish competence in implementing a particular intervention.
Researchers should consider training, supervision, and direct assessment of relevant skills.
Where substantial discretion is permitted, clear decision rules and provider preparation become especially important.
How Can Researchers Prevent Uncontrolled Intervention Drift?
Prevention begins with a sufficiently detailed intervention specification.
The protocol should identify essential functions, permitted modifications, delivery responsibilities, and procedures for reviewing consequential changes.
Researchers may use periodic observations, provider records, and implementation meetings to identify emerging differences.
However, monitoring should not be designed solely to enforce identical delivery when adaptation is intended.
The objective is to detect changes that threaten the intervention's defining functions while permitting legitimate flexibility.
Corrective actions should be documented, especially when they change how the intervention is delivered during the study.
What Is the Role of an Adaptation Log?
An adaptation log records important changes made to intervention content or delivery.
Useful information may include what was changed, why, when, by whom, and whether the modification was planned or reactive.
Researchers may also document whether the modification was intended to preserve or alter an essential intervention function.
The FRAME framework provides a structured approach to describing intervention adaptations and modifications.
However, an adaptation log should not become an unnecessarily burdensome record of every minor variation.
Its purpose is to preserve information needed to understand the intervention actually delivered.
How Can Fidelity Be Assessed When Delivery Is Flexible?
Fidelity should be evaluated against the intervention's intended strategy, not against an assumption that every participant must receive identical activities.
For a flexible intervention, assessment may focus on whether essential functions were delivered and whether adaptations followed specified rules.
For example, an instructor may use different disciplinary examples while consistently requiring students to critique AI-generated research suggestions.
That variation may be consistent with fidelity.
By contrast, removing the critical evaluation activity may represent a consequential deviation.
Researchers should develop delivery fidelity measures that distinguish permissible adaptation from intervention drift.
Can Researchers Use One Fidelity Score for a Flexible Intervention?
A composite fidelity score may be useful for some purposes, but it can obscure important differences in intervention delivery.
For example, a provider might complete most optional activities while omitting one essential component.
A high overall percentage would not necessarily indicate that the intervention's intended mechanism was preserved.
Researchers should consider reporting component-level adherence, delivery quality, and adaptation information separately.
Any weighting of components should have a defensible rationale and should not be selected retrospectively to explain favorable results.
No universal fidelity percentage establishes that a flexible intervention was adequately implemented.
Can Excessive Flexibility Make a Null Result Difficult to Interpret?
Yes. If participants received substantially different versions of the intervention, a small or statistically nonsignificant average effect may be difficult to interpret mechanistically.
Some versions may have preserved the intended functions while others did not.
However, researchers should not automatically conclude that poor implementation explains a null result.
The intervention may genuinely be ineffective, the outcome may be insensitive, or the study may lack sufficient precision.
Fidelity and adaptation data can help distinguish plausible explanations, but they do not independently establish what would have happened under perfect implementation.
The broader distinction between intervention failure and implementation failure requires additional evidence.
Can Excessive Flexibility Make a Positive Result Difficult to Explain?
Yes. A positive average effect may indicate that the overall intervention strategy was beneficial under the studied conditions.
However, if providers delivered different versions, the result may not identify which components or adaptations produced the benefit.
For example, improvement might be associated with structured feedback at some sites and additional tutoring at others.
Without suitable component-level evidence or a design intended to compare versions, researchers cannot confidently attribute the effect to a particular activity.
They should distinguish evidence for the intervention package from evidence for individual mechanisms.
How Does Excessive Flexibility Affect Replication?
Replication requires sufficient information about the intervention strategy to reproduce its defining characteristics.
If a study reports only that providers delivered individualized support, future researchers may not know which activities, decision rules, or adaptations were permitted.
Two subsequent studies could use the same intervention label while evaluating substantially different treatments.
Detailed intervention descriptions, adaptation rules, and implementation records can reduce this ambiguity.
The TIDieR checklist provides a framework for describing intervention materials, procedures, providers, delivery, tailoring, modifications, and fidelity.
Replication does not always require identical delivery, but it requires clarity about what constitutes the intervention.
Does Flexibility Affect Generalizability?
Potentially. A flexible intervention may be more feasible across diverse settings, but its applicability depends on whether the essential strategy can be reproduced.
If adaptation rules are explicit, future implementers may be able to tailor delivery while preserving the intervention's intended functions.
If flexibility is unrestricted, different institutions may implement incompatible versions.
The resulting evidence may be difficult to transport to a particular target setting.
Researchers should therefore consider both the applicability costs of excessive standardization and the interpretive costs of excessive flexibility.
How Does Flexibility Relate to the Target Estimand?
The estimand specifies the treatment effect the study intends to estimate.
If the treatment strategy is defined as a flexible program with explicit adaptation rules, variation within those rules may be part of the intended intervention.
If providers introduce modifications outside those rules, the actual treatment may depart from the specified strategy.
Researchers should clarify whether they are estimating the effect of offering a flexible intervention package, a fixed version, or a particular adaptive decision strategy.
These questions are not interchangeable.
The ICH E9(R1) framework offers a useful approach to defining treatment conditions and relevant intercurrent events, although its application should reflect the research context.
Should Researchers Analyze Outcomes by Intervention Version?
Version-specific analyses may provide useful information about implementation heterogeneity.
However, versions may not have been randomly assigned. Providers may choose adaptations according to participant needs, site resources, or early responses.
Consequently, observed differences between versions may reflect confounding or selection rather than causal effects of the modifications.
Researchers should avoid interpreting exploratory comparisons as definitive evidence that one version is superior.
If comparing intervention versions is a central objective, an appropriately designed comparative study may be required.
When Should the Intervention Protocol Be Revised?
Repeated unplanned modifications may indicate that the original intervention specification is impractical or insufficiently responsive to context.
Researchers should investigate why providers are changing the intervention and whether those changes preserve essential functions.
Minor clarifications may be manageable within existing procedures. Substantive changes affecting treatment content, participant safety, or the intended comparison may require protocol amendment and appropriate oversight.
Changes should not be introduced simply because interim results appear disappointing or favorable.
Researchers should document when revisions occurred and which participants received different intervention versions.
How Should Intervention Flexibility Be Reported?
Reports should describe the intended intervention, permissible adaptations, and actual modifications observed during implementation.
Researchers should identify consequential differences across providers or sites and explain whether essential functions were preserved.
Where feasible, delivery fidelity and adaptation data should be reported alongside the primary outcomes.
However, implementation findings should not be presented as proof that particular adaptations caused the observed treatment effects.
Transparent reporting helps readers understand what was tested, what varied, and which conclusions the evidence can reasonably support.
