03 · What You Need to Know
How Inclusion Interacts With Different Forms of Scientific Validity
What Does Inclusive Research Design Mean Methodologically?
Inclusive research design involves considering whether the people, experiences, and circumstances relevant to a research question can meaningfully contribute to the evidence being collected.
It may involve eligibility criteria, sampling frames, recruitment channels, accessible instruments, communication arrangements, research locations, or the demands placed on participants.
Inclusion is not equivalent to admitting every possible participant. A study investigating a defined clinical condition, educational setting, or developmental stage may require legitimate population boundaries.
The methodological objective is to avoid exclusions that are unnecessary for answering the research question and that could undermine the intended inference.
Researchers should begin by identifying which population groups may be systematically excluded by the proposed design and why those exclusions matter.
Which Forms of Validity Can Inclusive Design Affect?
Scientific validity encompasses several related but distinct concerns. Inclusion may influence each differently.
| Validity concern |
Relevant question |
Potential contribution of inclusion |
| Internal validity |
Does the study support the intended causal or other within-study inference without consequential bias? |
May reduce certain selection problems, but inclusion alone does not establish causal identification |
| External validity |
Can findings reasonably apply beyond the studied participants and conditions? |
May improve applicability when relevant population characteristics and contexts are represented |
| Construct validity |
Do the measures and procedures adequately represent the intended concepts? |
May reveal whether instruments function differently across relevant groups |
| Statistical conclusion validity |
Are statistical inferences supported by the data, assumptions, and analytical procedures? |
May improve information for subgroup analyses when sampling and analysis are appropriately planned |
| Qualitative credibility and transferability |
Do interpretations adequately reflect the phenomenon, and is sufficient context available to assess relevance elsewhere? |
May incorporate experiences or contextual variation essential to interpretation |
These categories are useful analytical distinctions rather than independent guarantees of research quality. A design decision may strengthen one aspect of validity while creating challenges for another.
When Does Inclusion Improve External Validity?
External validity concerns the applicability of findings to populations, settings, interventions, or circumstances beyond those directly studied.
Inclusive design may improve external validity when the original study population omits characteristics that influence the outcome or modify the effect of an intervention.
For example, an intervention intended for adults across a broad age range may produce evidence of limited applicability if the study excludes older adults without a relevant scientific justification.
Including older participants may provide information about whether the intervention operates similarly across age groups. However, the presence of older participants does not automatically establish precise age-specific effects or guarantee generalizability.
External validity also depends on research settings, implementation conditions, outcome definitions, and the relationship between the studied and target populations.
The CONSORT 2025 explanation and elaboration document emphasizes that trial settings, eligibility criteria, and intervention details help readers judge applicability. It also distinguishes the validity of the trial's findings from judgments about where those findings can reasonably be applied.
Does Inclusive Design Automatically Improve Internal Validity?
No. Internal validity depends on whether the study design and conduct support the intended inference within the population and conditions studied.
In a randomized trial, for example, random assignment can support causal inference within the randomized population when relevant assumptions and implementation requirements are satisfied. Expanding eligibility does not automatically improve the validity of that randomized comparison.
However, inclusion-related decisions can matter for internal validity when selection after assignment, differential attrition, or other participation mechanisms introduce bias.
In observational studies, restricting participation based on variables related to exposures and outcomes can sometimes induce selection bias. Broadening participation may reduce that problem under particular causal structures, but it is not a universal remedy.
Researchers should distinguish selection occurring before randomization from selection occurring afterward. Eligibility restrictions applied before random assignment primarily define the trial population and affect applicability; post-randomization exclusions can threaten the internal validity of the randomized comparison.
The relationship between inclusion and internal validity therefore depends on the study design, timing of selection, and causal assumptions.
How Can Inclusive Sampling Reduce Coverage Error?
Coverage error occurs when the sampling or recruitment mechanism does not adequately include the intended population.
Suppose a researcher wants to estimate satisfaction with public services among all residents but recruits exclusively through an online portal. Residents who do not use the portal cannot enter the sample through that channel.
Introducing additional recruitment methods may improve coverage. Whether this reduces bias depends on the relationship between portal use, participation, and the outcomes being estimated.
Researchers should distinguish improved coverage from statistical representativeness. Adding recruitment channels does not automatically establish known selection probabilities or eliminate differential nonresponse.
Where subgroup estimates are important, oversampling selected population groups may improve precision, provided the sampling design and analysis support the intended inference.
When Does Inclusion Improve Measurement Validity?
Measurement validity concerns whether the evidence supports the intended interpretation of a measure or instrument.
Inclusive design may reveal problems that remain unnoticed when an instrument is tested only among a narrow participant group.
For example, a questionnaire developed among highly digitally proficient university students may contain terminology or interaction requirements that function differently among participants with limited digital experience.
Including relevant users during instrument development and testing can help identify ambiguous wording, inaccessible interfaces, or differences in response processes.
However, broadening participation without adapting or evaluating the instrument can also introduce measurement error.
Researchers should assess whether research materials accommodate different literacy and disability-related needs while preserving the constructs being measured.
Where scores are compared across groups, evidence concerning measurement invariance or other relevant comparability properties may be necessary.
Can Including More Heterogeneous Participants Reduce Statistical Precision?
It can under some circumstances. Broader eligibility may introduce greater variability in participant characteristics, baseline outcomes, or responses to an intervention.
Greater outcome variability can reduce precision for a given sample size, although the actual consequences depend on the design, analysis, covariate adjustment, and estimand.
Researchers should not assume that heterogeneous samples are inherently inferior. Variation may be scientifically important, particularly when the intervention is intended for a diverse population.
The appropriate response may involve increasing sample size, stratifying randomization, using justified covariate adjustment, or prespecifying subgroup analyses rather than excluding relevant participants merely to reduce variability.
Nevertheless, these strategies require appropriate planning and may not be feasible for every study.
Why Does Effect Heterogeneity Matter for Inclusive Intervention Research?
Effect heterogeneity occurs when an intervention's effect differs across participants, settings, or circumstances.
If relevant effect modifiers are absent from the study population, the estimated average effect may not adequately describe what would happen in a broader target population.
For example, an educational intervention requiring frequent internet access may be effective among students with reliable connectivity but less effective among students with unstable access.
Including students with different connectivity circumstances may help researchers understand how the intervention performs across the intended population.
However, simply enrolling diverse participants does not guarantee sufficient statistical power to estimate subgroup-specific effects.
Researchers should prespecify important hypotheses where possible and avoid interpreting differences in subgroup statistical significance as proof of effect modification.
Can Inclusive Design Improve Qualitative Research?
Yes, when inclusion helps capture experiences, contexts, or perspectives essential to the research question.
Qualitative research does not generally require samples that reproduce population proportions. Its sampling logic may instead emphasize conceptual relevance, variation, information richness, or theoretical development.
For example, a study investigating barriers to accessing educational support may produce an incomplete account if it recruits only students who successfully use the support service.
Including students who attempted but could not access the service may reveal processes absent from the original sample.
However, adding participants from every demographic category is not automatically necessary. The appropriate sampling strategy depends on the qualitative approach and the phenomenon under investigation.
Researchers should explain how participant selection supports the intended interpretation and acknowledge perspectives that remain outside the study.
How Does Inclusive Design Relate to Research Ethics?
Scientific validity and research ethics are related but distinct. A methodologically valid study may still raise ethical concerns if participation burdens are unfairly distributed or relevant groups are excluded without adequate justification.
Conversely, an inclusive recruitment policy does not establish scientific validity if the study lacks appropriate measurements or an adequate analytical strategy.
The World Medical Association's Declaration of Helsinki emphasizes the ethical responsibilities involved in research with human participants, including appropriate consideration of participant protection and fair inclusion.
In the United States, NIH's Inclusion Across the Lifespan policy requires NIH-supported human-subjects research to include participants of all ages unless scientific or ethical reasons justify exclusion. This is a specific funder policy rather than a universal rule for all research.
Researchers should distinguish ethical obligations, institutional requirements, and methodological recommendations while recognizing that these considerations may influence the same design decisions.
When Might Broader Inclusion Compromise Research Quality or Participant Safety?
Broader inclusion may be inappropriate when participants face unacceptable study-related risks, when the research question requires a clearly defined population, or when available procedures cannot produce interpretable measurements for a proposed group.
For example, an intervention with a documented contraindication may require excluding participants with a particular clinical characteristic.
Similarly, a study measuring unaided visual performance may require specific functional conditions because those conditions define the intended measurement.
Researchers should avoid using broad demographic categories as substitutes for more precise scientific or safety criteria.
The relevant question is whether eligibility restrictions are necessary and proportionate to the research objective and participant risks.
Where accommodations or additional safeguards can support participation without undermining validity, they should be considered. Where inclusion would compromise safety or the intended measurement, a justified restriction may be appropriate.
Does Inclusive Design Guarantee Generalizability?
No. A sample can include participants from several demographic groups while remaining unrepresentative of the target population.
Generalizability depends on how participants were selected, the characteristics of the studied population, the research setting, relevant effect modifiers, and assumptions connecting the study to the population of interest.
For probability-based descriptive research, coverage, selection probabilities, nonresponse, and weighting may be central.
For causal research, transporting an estimated effect to another population may require assumptions about effect modifiers, intervention consistency, and the comparability of relevant conditions.
For qualitative research, transferability depends partly on the depth of contextual description and the relationship between the studied circumstances and those to which readers may apply the findings.
Inclusive design can strengthen these foundations, but it cannot replace the methodological reasoning required for each form of inference.
How Should Researchers Evaluate Whether Inclusion Improved Validity?
Researchers should identify the validity concern before introducing an inclusion-related modification.
If the concern is coverage error, evaluate whether the revised recruitment strategy reaches population segments previously absent from the frame.
If the concern is measurement, assess whether revised materials improve comprehension and usability without changing the construct.
If the concern is external validity, examine whether the included participants and settings better reflect the relevant target population and whether important effect modifiers are represented.
If the concern is differential attrition, monitor whether revised participation arrangements reduce missingness and whether remaining missing-data mechanisms are adequately addressed.
Researchers should avoid claiming that inclusion improved validity solely because demographic diversity increased. The claim should be supported by evidence concerning the particular inferential problem.
Watch Out
Do not equate demographic diversity with scientific validity. Inclusion improves validity only when it addresses a relevant limitation and the resulting design preserves the assumptions necessary for defensible inference. A broader sample can still contain serious selection, measurement, or analytical problems.
