Can Local Evidence Be More Useful for a Local Decision Than Stronger Evidence From a Very Different Setting?

What Should You Do When the Best Study Is Not the Closest Match to Your Setting?

Imagine that you need to make a local decision. One source of evidence is a rigorous randomized trial, systematic review, or other strong body of research conducted in settings substantially different from yours. Another is a smaller local study that resembles your population, institutions, resources, and implementation conditions much more closely but has methodological limitations.

Which evidence should matter more?

This is not adequately answered by choosing either “the strongest study” or “the local study.” Methodological credibility and applicability to a particular decision are different dimensions. Strong evidence can be indirect for your question, while highly applicable local evidence can still be seriously biased. Good decision-making requires examining both.

Local Evidence Can Be More Decision-Relevant, but It Does Not Automatically Become More Trustworthy

Evidence Quality and Evidence Applicability Answer Different Questions

Strong evidence can be strong for one question and indirect for another

Evidence quality is not simply a property that determines usefulness everywhere.

A well-conducted randomized trial may provide highly credible evidence about the effect of an intervention in the population and setting studied. If your decision concerns a substantially different population, delivery model, comparator, outcome, or context, the trial may provide less direct evidence for your particular question.

GRADE explicitly treats indirectness as a distinct consideration in assessing certainty. Indirectness can arise when the population, intervention, comparator, or outcomes in the available evidence differ from those specified in the question of interest.

AHRQ's applicability guidance makes a related point: evidence may provide strong support for decisions in one setting while being less applicable to another. Applicability depends on the question and the needs of the evidence user rather than on study design alone.

Neither dimension can simply substitute for the other.

Local evidence is especially useful for parameters that are inherently local

Some information is expected to vary substantially across settings and therefore often requires local data.

Examples include disease prevalence, baseline educational performance, service utilization, staffing, intervention uptake, technology access, local costs, transportation barriers, implementation capacity, policy compliance, and population preferences.

A large international randomized trial may be an excellent source for estimating an intervention's relative effect while telling a local hospital almost nothing about how many eligible patients it has, what implementation would cost, or whether enough personnel are available.

A modest local dataset may be much more useful for those particular questions.

Do not ask which study is “better” without asking better for what

Suppose a rigorous international trial and a local observational study appear to compete. The instinct may be to rank them and select one.

Often they are answering different parts of the decision.

The international trial may provide the more credible estimate of causal effect. The local observational data may provide the more credible estimate of baseline risk or current practice. Administrative data may establish how many people would be eligible. A local costing study may estimate the required budget.

The decision can use all of these sources for the parameters they are best suited to inform.

This avoids forcing one study to answer questions it was never designed to answer.

Baseline risk can make the same relative effect produce different local benefits

AHRQ's applicability framework highlights underlying or baseline risk as a major determinant of applicability. Even when relative effects remain reasonably constant, absolute benefits and harms can vary when baseline event rates differ.

Consider a hypothetical intervention that reduces the relative probability of an undesirable outcome by 20%.

The relative effect is identical in this hypothetical example, but the expected absolute benefit differs substantially. Local baseline data can therefore be essential for deciding whether an intervention is worthwhile.

This does not mean the local data provide better evidence of the intervention's causal effect. They provide a parameter needed to translate that effect into the local decision.

Local implementation conditions can change what external evidence means in practice

An intervention tested with specialist staff, reliable infrastructure, intensive training, or substantial implementation support may not be delivered identically in routine local practice.

Local evidence about staffing, workflows, infrastructure, adherence, fidelity, reach, and organizational readiness can therefore be essential for estimating what is realistically achievable.

This is one reason resource differences can create legitimate local evidence needs even when the underlying effect has already been studied rigorously elsewhere.

Local evidence can be highly applicable and still badly biased

This is the central caution.

A study does not become trustworthy simply because its participants live nearby. A small convenience sample, uncontrolled before-and-after study, poorly measured survey, or confounded observational analysis retains those limitations regardless of geographical relevance.

Local relevance cannot repair selection bias, confounding, measurement error, inadequate comparison groups, missing data, or inappropriate analysis.

When stronger external evidence and weaker local evidence appear to conflict, do not automatically choose the local result. First ask whether the difference is credible or whether methodological limitations provide a more plausible explanation.

Strong external evidence should not be discarded because the setting is different

Differences between settings do not automatically invalidate external evidence.

Ask whether those differences plausibly modify the relevant effect or simply affect other decision parameters. If an intervention's mechanism is expected to operate similarly and findings have already been consistent across diverse settings, external evidence may remain highly informative.

The question is whether the contextual difference is scientifically meaningful for the inference being transferred.

A vague assertion that “our context is different” should not outweigh a rigorous evidence base.

Evidence synthesis can be more useful than choosing a winner

Decision-making often improves when evidence is integrated rather than treated as competing camps.

A systematic review may estimate relative effectiveness. Local surveillance can estimate baseline risk. Administrative records can establish the size of the eligible population. Local implementation studies can identify staffing requirements and barriers. Economic analysis can estimate costs under local prices.

Together, these sources can support a more directly applicable decision than any one source alone.

AHRQ has explicitly explored how healthcare-system data can augment systematic-review findings by filling local evidence gaps and examining applicability to real-world populations. The underlying logic is useful beyond healthcare: local data can complement rather than replace stronger external evidence.

The local study may be more useful for the decision without being stronger science overall

This distinction is subtle but important.

A local survey showing that only 40% of a target population currently has access to a required technology may be methodologically modest compared with a large randomized trial demonstrating that the technology-supported intervention improves outcomes. Yet the local survey may be essential to deciding whether immediate implementation is feasible.

It would be misleading to say that the survey provides “stronger evidence” than the trial. It provides more directly relevant evidence for one component of the local decision.

Precision in language prevents applicability from being confused with methodological superiority.

Sometimes local evidence should challenge the assumption of transferability

Local data can also reveal that external evidence may not transfer as expected.

Suppose an intervention repeatedly succeeds internationally, but local implementation data show that a component essential to its mechanism cannot be delivered under current conditions. Or local observational evidence suggests an unexpectedly different baseline pattern that materially changes expected absolute benefit.

These findings can justify further investigation rather than blind application of the external estimate.

The appropriate response depends on the uncertainty. Sometimes focused local research is enough. Sometimes a full local effectiveness study becomes warranted. This is part of deciding when international evidence still leaves a necessary local research question.

Decision stakes affect how much local uncertainty you can tolerate

The consequences of being wrong matter.

A low-cost, reversible intervention with minimal harms may reasonably be adopted using strong external evidence even when some local uncertainty remains. A costly, difficult-to-reverse policy with substantial potential harms may warrant stronger evidence about local applicability before implementation.

This does not create a universal rule requiring local trials for high-stakes decisions. It means that the value of reducing local uncertainty increases when the consequences of a mistaken decision are large.

Local decision-makers may need evidence external studies never intended to provide

A ministry, hospital, university, or community organization often asks practical questions that differ from the primary research question in published studies.

How many people will qualify? What will implementation cost? Which subgroup is least likely to be reached? Can existing staff deliver the intervention? What infrastructure must be added? How will the intervention interact with current policy?

These are not inferior questions. They are often the questions that determine whether evidence can become action.

Sometimes no new local primary study is necessary

The existence of a local evidence need does not automatically require a new research project.

Administrative databases, registries, routine monitoring, national surveys, institutional records, or existing observational datasets may already provide the local parameters needed for the decision.

Before commissioning another study, identify the missing parameter and determine whether credible data already exist. The best evidence strategy is the one that resolves the decision uncertainty efficiently and defensibly, not necessarily the one that generates the newest dataset.

Combining Strong External Evidence With Relevant Local Evidence

Common Mistakes When Comparing Local and External Evidence

Match Each Evidence Source to the Part of the Decision It Can Best Inform

Instead of ranking whole studies and asking which one wins, decompose the decision. What do you need to know about causal effects, baseline conditions, population size, resources, costs, preferences, implementation, and harms?

Then identify the strongest sufficiently applicable evidence for each question.

The objective is not to defend local evidence against international evidence. It is to construct the most credible evidence base for the actual decision being made.

Before Choosing Between Local and External Evidence

Questions About Local and External Evidence

Use the Most Credible Evidence for Each Part of the Local Decision

How to Cite This Guide

This guide is intended to be read, shared, and used in research, teaching, and academic work. If you draw on its ideas, explanations, or other content, please acknowledge the source by citing the guide. Doing so gives appropriate credit and helps your readers locate the original resource.