Should Research Studies Reflect How an Intervention Will Be Used in Real-World Practice?

Should an Intervention Be Studied Under the Conditions in Which People Will Actually Use It?

A research team may demonstrate that an intervention improves outcomes when delivered by specially trained personnel, supported by additional resources, and monitored under carefully controlled conditions. The findings may be scientifically credible, yet practitioners may struggle to reproduce the intervention outside the study.

Consider an educational intervention that improves student performance when participants receive individualized technical assistance, frequent reminders, and additional instructional support. If ordinary schools or universities cannot provide those resources, the study may offer limited evidence about what would happen during routine implementation.

Should researchers therefore design intervention studies to resemble everyday practice from the beginning? Or are controlled conditions sometimes necessary to determine whether an intervention has a genuine effect? The answer depends on the scientific question and the decisions the resulting evidence is intended to inform.

Real-World Conditions Should Shape the Design When Routine Effectiveness Is the Research Objective

Designing Intervention Research That Produces Applicable Evidence

What Does Real-World Intervention Use Mean?

Real-world intervention use refers to the delivery and experience of an intervention under conditions resembling those in which it would ordinarily be implemented outside a research project.

These conditions may include routine staffing, existing infrastructure, typical participant characteristics, competing organizational demands, ordinary levels of adherence, and the resources normally available to practitioners.

For example, a digital learning intervention intended for university courses may ultimately be delivered by regular instructors using existing learning management systems. A study conducted exclusively with specialist facilitators, premium equipment, and intensive researcher assistance may evaluate a different delivery arrangement.

Real-world conditions are not necessarily uniform. Universities, schools, clinics, and communities differ substantially in resources, organizational practices, and participant needs. Researchers should specify which settings and implementation circumstances their study is intended to represent.

What Is the Difference Between Explanatory and Pragmatic Intervention Research?

The distinction between explanatory and pragmatic approaches was articulated by Schwartz and Lellouch in 1967. It remains influential in contemporary trial methodology.

An explanatory approach emphasizes whether an intervention can produce an effect under conditions selected to clarify efficacy or mechanisms. A pragmatic approach emphasizes the consequences of choosing an intervention under conditions relevant to ordinary practice.

These approaches are not mutually exclusive categories. Individual design features can be more explanatory or more pragmatic depending on the research objective.

Neither approach is inherently superior. The appropriate design depends on whether the intended conclusion concerns efficacy under defined conditions or effectiveness in practice. Loudon and colleagues' PRECIS-2 methodology explicitly frames this distinction as a matter of designing trials that are fit for purpose.

Why Can a Successful Controlled Trial Produce Disappointing Results in Practice?

An intervention may depend on conditions that are difficult to reproduce outside the research setting.

These conditions may include highly experienced providers, unusually motivated participants, specialized equipment, additional supervision, or resources unavailable to ordinary organizations.

For example, an educational intervention may appear effective when researchers provide individual coaching and continuous troubleshooting. If the intended implementation model assumes that one instructor will support a large class without additional personnel, the original findings may not establish effectiveness under that arrangement.

This does not mean that the controlled study was invalid. It means that its results describe an intervention delivered under particular conditions.

Researchers should examine whether the intervention is too highly controlled to implement in ordinary practice before making broad recommendations.

How Does the Intended Research Question Determine the Appropriate Level of Realism?

The question should establish whether the study is primarily concerned with intervention efficacy, effectiveness, mechanisms, implementation, or some combination of these objectives.

If the purpose is to determine whether a new instructional strategy can improve a specific cognitive process, controlled delivery may help isolate the relevant mechanism.

If the purpose is to determine whether universities should adopt the strategy, routine staffing, student access, scheduling, and implementation conditions become more consequential.

A study can also address both scientific interests, although this may require additional measurements, sample size, resources, or a more complex design.

The central problem is not using controlled conditions. It is drawing conclusions about routine effectiveness without sufficient evidence that the studied conditions resemble those under which the intervention would be implemented.

What Is the PRECIS-2 Framework?

The Pragmatic-Explanatory Continuum Indicator Summary 2, known as PRECIS-2, is a tool developed to help researchers assess whether trial design decisions are consistent with the intended pragmatic or explanatory purpose.

It evaluates nine domains: eligibility, recruitment, setting, organization, flexibility of delivery, flexibility of adherence, follow-up, primary outcome, and primary analysis.

Each domain is scored from 1, representing a very explanatory approach, to 5, representing a very pragmatic approach. The scores support discussion of design choices rather than functioning as a universal quality rating.

A trial does not become scientifically better merely because every domain receives a high pragmatic score. The authors emphasize that the tool helps investigators match their design to the purpose of the trial.

PRECIS-2 was developed for randomized trials. Its underlying questions may inform broader intervention research, but its formal scoring should not be assumed to have been validated for every nonrandomized or qualitative design.

Should Study Participants Resemble the Intended Users?

When the research objective concerns effectiveness in routine practice, participants should generally reflect the people expected to receive the intervention.

Restrictive eligibility criteria can limit applicability if they exclude characteristics common among intended users. For example, a digital learning intervention intended for working adults may produce incomplete evidence if the study recruits only full-time students with reliable connectivity and flexible schedules.

However, eligibility restrictions may be necessary for safety, scientific coherence, or the intended intervention.

Researchers should evaluate whether eligibility criteria unnecessarily exclude relevant population groups while preserving justified restrictions.

Including a broader population also does not automatically guarantee representativeness or sufficient precision for subgroup comparisons.

Should Interventions Be Delivered by Ordinary Practitioners?

When routine effectiveness is the primary objective, intervention delivery should reflect the expertise and responsibilities of the practitioners who would ordinarily implement it.

For example, an educational intervention intended for regular faculty members may need evaluation under conditions involving typical teaching workloads, institutional training, and available instructional resources.

Nevertheless, practitioners may require training to deliver an intervention competently. A pragmatic study does not require withholding preparation that would be part of responsible routine implementation.

The relevant distinction is between resources that would reasonably accompany adoption and extraordinary research-specific support unlikely to be available afterward.

How Much Variation in Intervention Delivery Should Be Allowed?

Routine practice often involves variation across providers, organizations, and participant circumstances.

Allowing appropriate variation can improve the relevance of effectiveness evidence, but uncontrolled differences may make it difficult to determine what intervention was actually delivered.

Researchers should identify essential intervention functions and distinguish them from components that may be adapted to local circumstances.

For example, a learning intervention may require timely formative feedback while allowing instructors to select different communication channels suited to their courses.

Determining which intervention components are essential and which can be adapted helps preserve interpretability without imposing unnecessary uniformity.

Does Real-World Research Require Randomization?

No. Randomized pragmatic trials are one important approach, but real-world intervention research can also use quasi-experimental, observational, mixed-methods, or implementation-focused designs.

The choice depends on the research question, feasibility of assignment, available comparison conditions, and the assumptions needed for causal inference.

Randomization may remain feasible in ordinary practice settings, including through cluster randomization where organizations or groups are assigned to intervention conditions.

However, studying an intervention in a routine setting does not automatically eliminate confounding or selection bias.

Researchers should distinguish the realism of the setting from the strength of the design used to support causal conclusions.

Which Outcomes Should Be Measured in Real-World Intervention Studies?

Outcomes should reflect the decisions the evidence is intended to inform.

For educational interventions, relevant outcomes may include learning performance, course completion, sustained engagement, or other measures aligned with the intervention's purpose.

For healthcare interventions, outcomes may include symptoms, functioning, quality of life, or clinically important events.

Intermediate measures can help explain mechanisms, but improvements in process indicators do not necessarily establish meaningful benefits for intended users.

Researchers should also consider whether implementation outcomes should be measured alongside effectiveness when adoption, delivery, or sustainability is central to the decision.

How Can Real-World Conditions Affect Intervention Effects?

Intervention effects may vary with organizational resources, provider expertise, participant characteristics, or implementation conditions.

For example, a learning support intervention may operate differently in institutions with strong digital infrastructure than in institutions where students experience frequent connectivity interruptions.

These contextual differences may influence reach, intervention exposure, fidelity, and outcomes.

However, observing different results across settings does not automatically establish why they differ. Researchers should identify plausible effect modifiers and consider appropriate comparisons or analyses.

The role of implementation context in explaining intervention success or failure should be incorporated when it is relevant to the research question.

Can a Real-World Study Preserve Internal Validity?

Yes. Pragmatic design does not require abandoning randomization, appropriate comparison groups, prespecified outcomes, or sound statistical analysis.

The Patient-Centered Outcomes Research Institute's guidance on pragmatic research emphasizes balancing naturalistic study conditions with explicit attention to intervention fidelity, adherence, and monitoring.

Researchers should define what is being compared, document consequential variations, and account for the design's clustering, missingness, and other relevant features.

Greater resemblance to routine practice may improve applicability while introducing heterogeneity or operational challenges. These trade-offs should be addressed through design and analysis rather than treated as reasons to avoid pragmatic research.

Should Implementation Costs and Resources Be Considered?

When a study is intended to inform adoption, decision-makers may need evidence about the resources required to deliver the intervention.

An intervention may improve outcomes but require staffing, equipment, or training that exceeds what some organizations can sustain.

Researchers should consider cost, acceptability, feasibility, and sustainability when these factors influence whether the intervention can be used in practice.

Not every efficacy study requires a full economic evaluation. The extent of resource measurement should reflect the intended decisions and available research capacity.

When Is a Highly Controlled Study More Appropriate?

Controlled conditions may be appropriate when the intervention is at an early stage of development, its mechanisms remain uncertain, or safety and feasibility have not yet been established.

For example, investigators developing a novel instructional method may first need to determine whether the method produces the intended learning response under well-specified conditions.

Later research may examine whether the intervention remains effective when delivered by ordinary instructors across varied institutions.

This sequence is not mandatory for every intervention. The appropriate progression depends on existing evidence, risk, and the questions requiring resolution.

How Can Researchers Evaluate the Applicability of Their Findings?

Researchers should compare the studied population, setting, intervention delivery, resources, and outcome measures with the conditions under which the intervention is intended to be used.

Important differences should be described rather than concealed through broad claims of generalizability.

Frameworks such as PRECIS-2 can support prospective design discussions, while RE-AIM provides concepts relevant to reach, effectiveness, adoption, implementation, and maintenance.

These frameworks address different purposes. Neither independently establishes causal validity or guarantees that results will transfer to every setting.

Designing a Digital Learning Intervention Study for Routine University Teaching

Suppose the routine-practice study produces a smaller estimated benefit than the initial controlled evaluation. That difference would not, by itself, prove that the intervention became less effective because researcher support was reduced.

Differences in participants, settings, comparison conditions, measurement, and study design could also contribute. Appropriate analysis and additional evidence would be needed to identify the explanation.

The example illustrates why real-world intervention research should be designed around the decision that practitioners or institutions actually face.

Common Misconceptions About Real-World Intervention Research

How to Decide Whether Your Intervention Study Should Be Pragmatic

Begin by identifying whether the intended contribution concerns efficacy under specified conditions, effectiveness during routine delivery, or both. Then assess how closely the proposed procedures should resemble the circumstances of eventual use.

Researchers should document the rationale for explanatory and pragmatic features rather than assume that every design choice must fall at one end of the continuum.

Check Whether Your Intervention Study Reflects Its Intended Use

Frequently Asked Questions About Real-World Intervention Studies

Real-World Intervention Research Should Match the Decision Its Findings Are Intended to Inform

How to Cite This Guide

This guide is intended to be read, shared, and used in research, teaching, and academic work. If you draw on its ideas, explanations, or other content, please acknowledge the source by citing the guide. Doing so gives appropriate credit and helps your readers locate the original resource.