Is Your Trial Testing Whether an Intervention Can Work Under Ideal Conditions or Does Work in Usual Practice?

Are You Asking Whether an Intervention Can Produce an Effect or Whether Introducing It Will Improve Ordinary Practice?

You have developed a digital intervention intended to improve students' research writing. In a pilot study, trained facilitators deliver the program with extensive support, students receive regular reminders, and every instructional activity is carefully monitored.

The results appear promising.

You now want to know whether universities should adopt the program. But ordinary instructors have different workloads, students vary in their participation, and institutions cannot necessarily reproduce the intensive support provided during the pilot.

Would another tightly controlled trial answer the adoption question? Or should the next evaluation examine what happens when the program is introduced under normal institutional conditions?

The distinction concerns efficacy and effectiveness. Both are important, but they refer to different questions about an intervention's effects and the conditions under which those effects are estimated.

Efficacy Concerns Effects Under Specified Favorable Conditions; Effectiveness Concerns Effects Under Conditions Relevant to Practice

Distinguishing Intervention Efficacy From Effectiveness

What Is Intervention Efficacy?

Intervention efficacy concerns whether a treatment produces an intended effect under specified conditions selected to clarify or support its performance.

These conditions may involve selected participants, trained providers, carefully specified treatment procedures, and relatively intensive implementation support.

For example, researchers may investigate whether an AI-assisted writing program improves students' ability to evaluate research arguments when delivered by specially trained facilitators according to a structured protocol.

The study may provide evidence that the intervention can produce a benefit under those conditions.

However, efficacy does not necessarily mean the intervention was delivered perfectly, and the conditions need not be literally ideal.

Researchers should describe the actual treatment and implementation arrangements rather than use ideal conditions as an undefined methodological label.

What Is Intervention Effectiveness?

Intervention effectiveness concerns whether a treatment strategy produces useful effects under conditions relevant to its intended application.

For example, universities may want to know whether adopting an AI-assisted writing program improves students' independently completed research proposals when ordinary instructors deliver it alongside existing academic services.

An effectiveness study may permit variation in participant adherence, provider practices, and institutional support when these reflect the intended implementation strategy.

However, effectiveness research does not require unrestricted intervention delivery or the absence of research controls.

Researchers must still define the intervention sufficiently, measure outcomes appropriately, and protect the credibility of the treatment comparison.

How Are Efficacy and Effectiveness Different?

These differences are tendencies, not rigid rules. A study may combine efficacy-oriented and effectiveness-oriented design features.

Is Efficacy the Same as an Explanatory Trial?

The concepts are closely related but not completely interchangeable.

Explanatory trials generally emphasize efficacy-oriented questions, while pragmatic trials generally emphasize effectiveness-oriented questions.

However, explanatory and pragmatic characteristics describe how trial design choices align with their purposes.

Efficacy and effectiveness concern the treatment effects being investigated and the conditions under which they are relevant.

Researchers should understand the distinction between explanatory and pragmatic trials without treating the terms as exact synonyms.

The target estimand and intervention conditions should be stated explicitly.

Does an Efficacy Trial Test Whether an Intervention Works Under Perfect Conditions?

Not necessarily. Perfect implementation is rarely attainable or even clearly definable.

Efficacy studies may use conditions designed to reduce implementation variation, but participants can still miss sessions, providers may make errors, and outcomes remain subject to uncertainty.

Researchers should specify what conditions were selected and why they are relevant.

For example, a study may use trained instructors and structured feedback procedures because those conditions are believed necessary to activate the intervention's intended mechanism.

Calling these conditions ideal should not substitute for a detailed intervention description.

Does an Effectiveness Trial Mean Researchers Stop Controlling Implementation?

No. Effectiveness trials can retain rigorous intervention specifications, randomization, outcome assessment, and analytical procedures.

The difference is that delivery arrangements may be selected to resemble the circumstances under which the intervention is intended to be used.

For example, ordinary university instructors may deliver a structured program while following clearly defined core components.

Researchers may monitor fidelity without providing implementation support beyond what the adoption strategy includes.

The objective is to evaluate a meaningful treatment strategy under relevant conditions, not to allow uncontrolled methodological errors.

Why Might Efficacy and Effectiveness Results Differ?

Differences may arise because studies involve different participants, providers, treatment doses, comparison conditions, adherence patterns, or contexts.

For example, a learning program delivered by specialist facilitators may produce a larger effect than the same program delivered by ordinary instructors with limited preparation.

However, a smaller effectiveness estimate is not inevitable.

Some interventions may perform similarly across settings, and others may benefit from adaptations or contextual features present in routine practice.

Differences between efficacy and effectiveness estimates should not automatically be attributed to poor implementation.

Establishing the reasons requires appropriate comparative evidence and attention to differences between studies.

What Is the Efficacy–Effectiveness Gap?

The efficacy–effectiveness gap describes differences that may occur between effects observed under selected research conditions and effects observed under routine implementation.

For example, an intervention may improve performance when participants receive intensive support but produce smaller benefits when implemented with ordinary resources.

However, the term should not imply that every intervention necessarily exhibits such a gap or that the gap always reflects implementation failure.

Differences may arise from populations, comparison conditions, outcome measurement, treatment strategies, or contextual factors.

Researchers should avoid calculating a simple difference between results from unrelated studies and treating it as a causally identified implementation effect.

How Does Participant Eligibility Affect the Question?

Efficacy-oriented trials may restrict eligibility to participants for whom the intervention is particularly suitable or for whom treatment delivery can be standardized.

Effectiveness-oriented trials may include a broader range of participants when the adoption decision concerns that broader population.

For example, a university may need evidence from students with different academic preparation, accessibility needs, and levels of digital literacy.

However, broader eligibility is not always appropriate.

Researchers should consider safety, feasibility, intervention requirements, and the intended target population.

Eligibility should reflect the research question rather than a general assumption that broader samples are always better.

How Does Provider Training Influence Efficacy and Effectiveness?

Provider training may be essential to intervention delivery, particularly when the treatment depends on specialized skills.

An efficacy-oriented study may provide extensive training and supervision to establish whether the intervention can produce an effect when delivered competently.

An effectiveness-oriented study may evaluate the training and support package that institutions could realistically provide.

However, removing necessary training could change the intervention rather than simply make it more pragmatic.

Researchers should identify the competence and resources required for faithful delivery.

Where provider skills matter, intervention provider competence may warrant direct assessment.

How Does Intervention Fidelity Affect the Distinction?

Intervention fidelity concerns whether the treatment was implemented according to its intended design.

In an efficacy-oriented study, fidelity may be assessed against relatively fixed procedures.

In an effectiveness-oriented study, fidelity may concern preservation of core functions and adherence to permitted adaptation rules.

Neither orientation requires researchers to ignore implementation quality.

However, fidelity should be evaluated against the treatment strategy actually intended.

Researchers should avoid treating ordinary variation as implementation failure when that variation is explicitly part of the intervention.

Should Usual Care Be Standardized in an Effectiveness Trial?

Not necessarily. Variation in usual care may be relevant to the practical decision being studied.

For example, universities may differ in tutoring services, academic advising, or instructional resources.

An effectiveness trial may evaluate the effect of introducing a new program within those existing systems.

However, researchers should describe usual care and consider whether important differences between treatment groups are consistent with the intended comparison.

Unplanned differential co-interventions may still complicate interpretation.

The appropriate approach depends on the target estimand and implementation setting.

What Role Does Participant Adherence Play?

Adherence concerns whether participants follow prescribed treatment activities.

An efficacy-oriented study may use intensive reminders, support, or monitoring to encourage treatment receipt.

An effectiveness-oriented study may allow adherence patterns resembling those expected during routine implementation.

However, adherence support may itself be part of the intervention package.

Researchers should distinguish the effect of offering a program with ordinary adherence from the effect that might occur under a specified level of treatment receipt.

These are different causal questions and may require different estimands and analytical methods.

Why Is the Estimand Important?

An estimand specifies the treatment effect or quantity that a study seeks to estimate.

It includes the treatment conditions, population, outcome, population-level summary, and approach to relevant intercurrent events.

For example, one study may seek the effect of assigning students to a digital program regardless of subsequent participation.

Another may seek the effect that would occur under a specified hypothetical adherence condition.

The two effects should not be treated as interchangeable.

The ICH E9(R1) framework provides a structured approach to defining these questions before selecting analytical methods.

Is Intention-to-Treat Analysis an Effectiveness Analysis?

Intention-to-treat analysis is commonly associated with estimating effects of randomized assignment, but the terminology requires care.

Analyzing participants according to assigned groups can preserve important benefits of randomization.

However, the estimand depends on the treatment strategy, intercurrent events, outcome definition, and analytical assumptions.

An intention-to-treat approach does not automatically make a trial pragmatic or establish real-world effectiveness.

Likewise, a per-protocol analysis does not automatically identify efficacy under perfect implementation.

Researchers should define the target effect rather than rely on analytical labels alone.

Can Per-Protocol Analysis Establish Efficacy?

Not automatically. Restricting analysis to participants who followed the intervention may introduce selection bias.

Participants who adhere may differ from nonadherent participants in motivation, prior ability, health status, or other outcome-related characteristics.

Consequently, comparing adherent participants across randomized groups may not preserve the original exchangeability established by randomization.

Estimating effects under specified adherence conditions may require methods that address relevant post-randomization confounding.

Researchers should not describe a naive per-protocol comparison as the intervention's true efficacy.

How Should Outcomes Be Selected?

The primary outcome should correspond to the intended effect and decision.

An efficacy-oriented study may prioritize a measure closely aligned with the proposed intervention mechanism.

For example, a research writing program may be evaluated using a structured task measuring critical evaluation of AI-generated arguments.

An effectiveness-oriented study may prioritize the quality of independently completed research proposals because that outcome matters to universities considering adoption.

However, these choices are not exclusive. A study may include both proximal mechanism measures and consequential outcomes.

Researchers should avoid assuming that a convenient assessment represents meaningful real-world improvement.

How Does Follow-Up Duration Affect Efficacy and Effectiveness?

Follow-up should reflect the expected timing of intervention effects and the decision the study intends to support.

An efficacy-oriented study may examine whether an intervention produces a short-term change in a targeted skill.

An effectiveness-oriented study may require longer follow-up to determine whether benefits persist under ordinary conditions.

For example, improvement immediately after a writing workshop does not necessarily establish sustained improvement in independent research writing.

However, longer follow-up is not automatically superior.

The appropriate period depends on the intervention mechanism, outcome, resources, and risk of missing data.

Does Effectiveness Research Have Better External Validity?

Effectiveness-oriented design may improve relevance to particular practice decisions when the study conditions resemble the intended implementation environment.

However, external validity is not guaranteed.

A trial conducted in a small number of well-resourced universities may have limited applicability to institutions with different infrastructure or student populations.

Researchers should describe the target population, participating settings, treatment delivery, and contextual conditions.

Claims about generalizability should be supported by the relationship between the study and target settings rather than the effectiveness label alone.

Can Efficacy and Effectiveness Be Studied in the Same Trial?

A trial may include both explanatory and pragmatic objectives, but researchers should distinguish the corresponding questions and analyses.

For example, a study might estimate the effect of introducing a digital program under ordinary conditions while also measuring whether specific learning mechanisms changed.

However, collecting mechanism measures does not automatically identify causal mediation.

Similarly, comparing outcomes among participants with different adherence levels does not automatically estimate an efficacy effect.

Researchers should prespecify the primary estimand and identify additional analyses as confirmatory or exploratory according to their design and evidential support.

Should an Efficacy Trial Always Come Before an Effectiveness Trial?

No universal sequence is required.

For some interventions, early efficacy studies may be appropriate because the causal theory, delivery procedures, or safety profile remains uncertain.

For others, existing evidence may already support the intervention sufficiently to justify evaluation under ordinary conditions.

Some complex interventions are developed and evaluated iteratively, with feasibility, implementation, and effectiveness questions informing one another.

The Medical Research Council framework emphasizes that intervention development and evaluation need not follow a rigid linear sequence.

Researchers should select the next study according to the most consequential unresolved question.

How Can Researchers Decide Which Effect Matters Most?

The choice should begin with the intended use of the evidence.

If the central uncertainty is whether the intervention can produce its hypothesized benefit under supported delivery, an efficacy-oriented evaluation may be appropriate.

If the central uncertainty is whether institutions should adopt the intervention under ordinary conditions, an effectiveness-oriented evaluation may be more informative.

Researchers should also consider whether the proposed delivery strategy is feasible, whether the intervention has a sufficiently developed evidence base, and which outcomes matter to decision-makers.

The choice should be reflected in eligibility, providers, comparison conditions, intervention flexibility, follow-up, and analysis.

Ultimately, the relevant question is not simply whether the intervention works, but for whom, under what conditions, and according to which treatment strategy.

From Supported Efficacy to Routine Effectiveness in an AI-Assisted Writing Program

Suppose the efficacy-oriented evaluation finds a substantial improvement in critical evaluation skills, while the effectiveness-oriented evaluation finds only a modest improvement in independent proposal quality.

The findings may reflect differences in outcomes, providers, treatment exposure, institutional conditions, or other features of the two evaluations.

They do not establish that ordinary instructors are less competent or that implementation failure caused the difference.

The appropriate conclusion is that the studies address different effects under different conditions. Further research may be needed to identify why the results differ and which implementation strategy best supports the intended outcome.

Common Misconceptions About Efficacy and Effectiveness Trials

Specify the Conditions Under Which You Want to Know Whether the Intervention Works

Before designing the trial, identify the decision that the findings should inform and the treatment strategy relevant to that decision.

Then determine whether the study should emphasize efficacy under selected conditions or effectiveness under circumstances resembling ordinary implementation.

Researchers should also consider how the trial's design choices align with the pragmatic–explanatory continuum and whether its intended implementation conditions are adequately represented.

The central question is not whether ideal conditions or usual practice are inherently preferable. It is which conditions are relevant to the effect the study needs to estimate.

Before Choosing an Efficacy- or Effectiveness-Oriented Evaluation

Frequently Asked Questions About Efficacy and Effectiveness Trials

Whether an Intervention Can Work Is Not Always the Same as Whether It Will Work in Practice

How to Cite This Guide

This guide is intended to be read, shared, and used in research, teaching, and academic work. If you draw on its ideas, explanations, or other content, please acknowledge the source by citing the guide. Doing so gives appropriate credit and helps your readers locate the original resource.