When Should Research Move From “Does It Work?” to “For Whom, When, and Why?”

When is the average effect no longer the most important question?

Early in the study of an intervention, program, treatment, or practice, asking whether it produces an effect is entirely reasonable. Researchers first need credible evidence that something happens at all.

As evidence accumulates, however, the average answer can become less informative. An intervention that produces a modest benefit on average might produce a substantial benefit for some people, little change for others, or different effects under different conditions.

At that point, continuing to ask only “Does it work?” can conceal the next scientifically important questions: for whom does it work, when does it work, under what conditions does it work, and why?

Move beyond the average effect when variation becomes the consequential uncertainty

An average effect can answer one question while hiding another

“Does it work?” usually asks about an average

Many intervention studies estimate an average effect across the participants included in the analysis. That estimate is useful. It tells researchers whether outcomes differ, on average, between relevant conditions under the assumptions of the design and analysis.

But an average does not imply that everyone experiences the average effect. Effects may vary across individuals, populations, settings, intervention characteristics, doses, durations, baseline risks, or other conditions.

This variation is often described as treatment-effect heterogeneity or, more generally, effect heterogeneity. A variable associated with differences in the effect may be described as an effect modifier or moderator, depending on the methodological framework being used.

Move toward heterogeneity when the existence of an average effect is no longer the main uncertainty

The transition becomes especially relevant when repeated credible studies or evidence syntheses support a reasonably stable overall conclusion. At that point, another study designed primarily to demonstrate the same average effect in essentially the same circumstances may provide diminishing information.

This does not mean the average effect becomes irrelevant. Rather, the research frontier may move. If new studies rarely change the overall conclusion, understanding meaningful variation can become more informative than repeatedly establishing the same central tendency.

“For whom?” asks whether effects differ across relevant people or groups

An intervention may not benefit all participants equally. Baseline characteristics, prior risk, age, disease severity, prior knowledge, socioeconomic conditions, or other features may be associated with different effects, depending on the field and question.

Researchers should be cautious here. Cochrane guidance emphasizes that subgroup analyses have substantial pitfalls. Conducting many post hoc subgroup comparisons can generate misleading apparent differences simply by chance. Comparing whether one subgroup has a statistically significant effect while another does not is also not a valid test that the effects differ between the groups.

Credible claims about differential effects therefore require appropriate interaction analyses, adequate information, preferably prespecified hypotheses, substantive plausibility, and careful interpretation.

“When?” asks about conditions and boundary effects

Variation does not have to be a property of people. Effects can depend on intervention intensity, duration, timing, setting, delivery mode, organizational conditions, comparator, follow-up period, or other contextual characteristics.

Identifying such boundary conditions can turn a broad statement such as “the intervention works” into a more useful statement about the circumstances in which a particular effect should be expected.

This can also explain apparent inconsistency in an accumulating literature. Studies that seem to disagree may actually have investigated meaningfully different conditions.

“Why?” can refer to mechanisms, but moderators and mechanisms are not the same

A moderator identifies variation in an effect. A mechanism concerns the process through which an exposure or intervention produces an outcome. These questions can be related, but they should not be collapsed into one another.

Suppose an educational intervention has a larger effect among students with lower baseline knowledge. Baseline knowledge may help identify for whom the intervention is particularly beneficial, but that observation alone does not explain the causal process through which the intervention improves learning.

When an association itself has become well established, researchers may need a more focused transition from association toward mechanism.

Heterogeneity should be explained carefully, not mined until something appears

Once researchers become interested in “for whom” and “when,” it is tempting to divide data into many subgroups and search for differences. That strategy can produce convincing-looking but unreliable findings.

Cochrane recommends caution when investigating heterogeneity through subgroup analyses and meta-regression. Analyses specified after inspecting the results are particularly vulnerable to spurious explanations and are better treated as hypothesis-generating. Even prespecified analyses require careful interpretation.

The shift toward heterogeneity therefore demands stronger questions, not simply more analyses.

A null average effect can also hide important heterogeneity

The move toward heterogeneity is not restricted to interventions with a positive average effect. An overall estimate near zero can sometimes arise because beneficial and harmful effects, or larger and smaller effects, occur under different conditions.

That possibility should not be used as an excuse to search indiscriminately for subgroups after an unpromising result. It becomes scientifically useful when there are credible reasons, suitable data, and appropriate methods for expecting differential effects.

The shift does not require waiting until everything about the average effect is certain

Research development is rarely perfectly sequential. Questions about heterogeneity and mechanisms can be planned while effectiveness evidence is still accumulating. Indeed, waiting until a long sequence of conventional studies is complete may waste opportunities to learn about context and variation.

The relevant issue is whether there is enough evidence and theoretical justification to support the more specific question. Research on effect modification requires adequate variation, measurement, sample size, and design. Asking a sophisticated question with data incapable of answering it merely upgrades the vocabulary, not the evidence.

The next transition may be toward real-world implementation

Once researchers understand that an intervention can produce benefits and have begun identifying important conditions surrounding those benefits, another question may become pressing: can the intervention be adopted, delivered, and sustained in routine settings?

That requires a different shift toward implementation research. Knowing for whom something works does not automatically tell you how to make it work in practice.

Moving beyond the average effect of a learning intervention

Another conventional effectiveness study could still contribute information. But if it simply reproduces the same average comparison in a familiar population, it may answer a question that the literature already addresses reasonably well while leaving the more consequential uncertainty untouched.

Moving beyond “Does it work?” requires more than adding subgroup analyses

Let the remaining uncertainty determine the next question

If the literature already provides credible evidence about an average effect, examine what researchers still cannot explain. Do outcomes vary meaningfully? Are important populations poorly represented? Are plausible effect modifiers known but inadequately tested? Is the mechanism uncertain?

The broader principle is simple: research questions should evolve with accumulated knowledge. Once a literature has repeatedly answered one question, novelty should not depend on asking it again with a slightly different sample. The more useful contribution may be identifying what becomes important once the original question is largely answered.

Decide whether the literature is ready for more specific questions

Questions about moving beyond average effects

Once the average answer is credible, variation may become the more important science

How to Cite This Guide

This guide is intended to be read, shared, and used in research, teaching, and academic work. If you draw on its ideas, explanations, or other content, please acknowledge the source by citing the guide. Doing so gives appropriate credit and helps your readers locate the original resource.