Can Inclusion Be Important Even When You Do Not Expect a Different Effect?

If you expect the same effect, why does inclusion still matter?

Suppose an intervention has been studied repeatedly and you have no strong theoretical reason to believe that its underlying effect will differ for an underrepresented population. Should researchers still make a deliberate effort to include that population?

It is easy to frame inclusion entirely as a search for differences. Under that logic, additional populations matter only when researchers expect a different coefficient, treatment effect, association, or outcome. If no difference is expected, inclusion can begin to look methodologically unnecessary.

That reasoning is too narrow. Research participants do more than provide opportunities to test effect heterogeneity. Their inclusion can determine whose experiences are represented, whether implementation barriers become visible, whether measurements function appropriately, and how confidently evidence can be applied to the people expected to use or be affected by it.

Inclusion can matter even when similar effects are entirely plausible

Inclusion serves purposes beyond finding population differences

Do not equate inclusion with subgroup comparison

Two research goals are often conflated. One is ensuring that relevant populations participate in research. Another is estimating whether an effect differs across those populations.

They are not the same methodological task.

A study can pursue the first without being designed or powered for the second. Conversely, simply recruiting participants from several populations does not guarantee that a study can estimate reliable population-specific effects.

This distinction matters because researchers sometimes feel compelled to invent an expected difference to make inclusion sound scientifically useful. There is no need. As discussed when considering whether another population should be expected to produce a different finding, uncertainty about applicability and the value of direct evidence can exist without a directional hypothesis.

Direct participation provides evidence about applicability

Generalizability concerns inference from a study sample to a defined target population. Methodological work in this area emphasizes that researchers must first specify the target population because a study cannot meaningfully be described as simply "generalizable" without saying to whom. A study may support inference to one population more readily than another.

This matters even when no effect difference is expected. If a population is part of the intended target population but is consistently absent from the evidence, researchers may have less direct empirical support for applying the findings to that population. A randomized trial can have strong internal validity for its participants while leaving a separate question about external validity.

Including relevant populations can reduce the inferential distance between the people who generated the evidence and the people to whom researchers, practitioners, or policymakers intend to apply it.

That does not mean every study sample must reproduce the demographic composition of society. Representativeness is meaningful only relative to a target population and an inferential goal. The important question is whether the people missing from the study create consequential uncertainty about the claim being made.

The effect may be similar while access to the intervention is not

Imagine that an educational technology improves learning when students use it as intended. Its underlying educational effect might be similar across populations. Yet students may differ substantially in whether they can access the technology, understand its interface, use required devices, obtain technical support, or engage with it under their actual learning conditions.

If researchers include only participants for whom these conditions are easy to satisfy, the study can answer "Does the intervention work when used?" while providing much less evidence about "Can the intended population realistically use it?"

The same distinction can appear in many fields. An intervention may have a comparable effect among those who receive it while recruitment, uptake, adherence, acceptability, accessibility, or implementation differs substantially across populations.

Inclusion can therefore reveal barriers that effect estimates alone may conceal.

Measurement may behave differently even when the phenomenon does not

Another reason for inclusion concerns measurement. Researchers may expect the underlying construct to operate similarly while remaining uncertain about whether their instrument captures it comparably.

A questionnaire may contain unfamiliar terminology. A digital assessment may impose accessibility barriers. A translated measure may not preserve the intended meaning of every item. A behavioral indicator may depend on opportunities that are unevenly available across settings.

These are not necessarily hypotheses about different underlying effects. They are questions about whether the research process itself provides valid and interpretable evidence for the population.

Population inclusion can expose such problems before researchers confidently extend conclusions beyond the circumstances in which the measures were originally developed or tested.

Similar findings can strengthen the scope of an existing conclusion

If there was a legitimate reason to question whether evidence extended to another relevant population, finding a similar effect is informative. It provides direct evidence that the finding may hold across the populations and conditions examined.

This should not be overstated. Similar findings in two populations do not prove universal generalizability. Nor does a nonsignificant subgroup difference prove that effects are identical. Generalizability depends on the target population, study design, relevant effect modifiers, and assumptions needed for the inference.

Still, direct evidence from a population can reduce uncertainty even when the result is reassuringly boring. Replication occasionally earns its keep without producing a dramatic interaction term.

Inclusion can matter because people are affected by the evidence

Research evidence often informs decisions beyond the study itself. Educational programs are adopted, health interventions are recommended, technologies are deployed, and policies are developed partly because research suggests that they work.

If a population is expected to experience those decisions, persistent exclusion raises a legitimate question: how much of the evidence supporting the decision was actually generated under conditions relevant to that population?

This does not establish that the effect must differ. Rather, it recognizes that the distribution of research participation and the distribution of research consequences can become disconnected.

Inclusion may therefore have an equity rationale in addition to an inferential one. The stronger argument is not that every demographic group must appear in every study. It is that populations materially affected by research-based decisions should not be systematically absent from the production of relevant evidence without a defensible reason.

That distinction becomes particularly important when asking whether studying another population improves fairness rather than merely adding another subgroup.

Inclusion should not become demographic box-checking

There is an important counterpoint. More demographic diversity does not automatically make a study methodologically stronger.

If a study recruits members of an underrepresented population but does not remove barriers to participation, collect relevant information, use appropriate measures, report participation transparently, or interpret findings responsibly, representation may remain largely symbolic.

Likewise, adding a small number of participants does not necessarily permit population-specific conclusions. Researchers should distinguish between broadening participation and producing reliable subgroup estimates.

Sometimes inclusion is preferable to creating another separate study

If researchers repeatedly discover that a population is missing and respond by creating isolated population-specific studies, the evidence base can become fragmented. A more inclusive design may sometimes be preferable.

For example, researchers might broaden recruitment sites, make procedures accessible, oversample populations that would otherwise remain too small for planned analyses, or deliberately collect characteristics relevant to generalizability. Which strategy is appropriate depends on the research question.

A separate study becomes more compelling when the population has distinctive research questions, contexts, methods, or priorities requiring sustained attention. That is why deciding whether an underrepresented population needs its own study is different from deciding whether it should be included in research at all.

Meaningful inclusion may require involvement before recruitment begins

Researchers can technically include a population while still designing a study around assumptions that make participation difficult or the questions unimportant to that population.

When accessibility, trust, terminology, community priorities, or locally meaningful outcomes are central to the research problem, participation in the final sample may be only one part of inclusion. Researchers may need to consider community involvement before the study is designed.

This is particularly relevant when conventional study procedures helped produce the underrepresentation in the first place.

The outcome can be similar while inclusion still changes what you learn

The important distinction is between the effect of an intervention under successful use and the conditions determining whether people can actually experience that intervention. Inclusion can matter greatly to the latter even when researchers expect similarity in the former.

Common mistakes when inclusion is discussed without expected differences

Specify what inclusion contributes before deciding how to achieve it

If you want to justify including an underrepresented population, do not begin by searching for a reason that its effect must differ. Begin by defining the target population and asking what would remain uncertain if the population continued to be absent.

The strongest rationale may therefore sound less dramatic than "we expect a different effect." It may simply state that a population is affected by the phenomenon or intervention, has been inadequately represented in the evidence, and needs direct inclusion to establish whether the evidence is applicable and implementable under the conditions that population actually experiences.

Before justifying inclusion without an expected different effect

Questions about inclusion when effects may be similar

Inclusion does not need a difference hypothesis to have research value

How to Cite This Guide

This guide is intended to be read, shared, and used in research, teaching, and academic work. If you draw on its ideas, explanations, or other content, please acknowledge the source by citing the guide. Doing so gives appropriate credit and helps your readers locate the original resource.