03 · What You Need to Know
How Eligibility Restrictions Shape the Meaning and Applicability of Research
What Are Eligibility Criteria?
Eligibility criteria specify which individuals or other units can participate in a study. They commonly include inclusion criteria, which identify characteristics required for participation, and exclusion criteria, which identify circumstances that make participation inappropriate or outside the study's intended scope.
In educational research, inclusion criteria might require enrollment in a particular course or academic level. Exclusion criteria might concern prior exposure to the intervention, inability to access essential study activities, or circumstances that make a specified assessment unsuitable.
In clinical research, eligibility may involve diagnosis, disease severity, treatment history, or safety-related considerations.
Eligibility criteria serve several legitimate purposes. They can establish a meaningful study population, protect participants, ensure that the intervention addresses the intended condition, and support interpretable outcome measurement.
However, every restriction also defines who is absent from the evidence. Researchers should consider whether those absences matter for the intended inference.
How Do Eligibility Criteria Affect the Target Population?
The target population is the population to which researchers intend to extend their findings.
The eligible population consists of individuals or units satisfying the study's specified participation criteria. These populations may differ substantially.
Suppose the target population includes all first-year engineering students at a university, but the study requires personal laptop ownership and uninterrupted internet access.
The eligible population becomes a subset of the intended population. Students without those resources cannot contribute direct evidence about intervention performance under their circumstances.
This does not necessarily make the study invalid. It means that the findings directly concern a narrower population and may require additional assumptions before being applied more broadly.
Researchers should distinguish the population they wish to inform from the population their recruitment procedures actually permit them to study.
Why Might Researchers Use Strict Eligibility Criteria?
Restrictions may be justified when participants would otherwise face unacceptable risks, when the intervention is not relevant to their circumstances, or when the study requires a clearly defined population to answer a specific question.
For example, an intervention designed to support students taking an introductory programming course may reasonably require enrollment in that course.
Likewise, a study evaluating a particular simulation may need participants to have completed prerequisite instruction because the intervention assumes knowledge not taught within the program.
Some restrictions are primarily operational. Researchers may exclude participants with irregular schedules, limited device access, or anticipated difficulty completing follow-up.
Operational restrictions can improve feasibility, but their methodological justification deserves scrutiny when they exclude common characteristics of the intended users.
Convenience alone does not establish that an exclusion is scientifically necessary.
Can Restrictive Criteria Improve Internal Validity?
Sometimes. Restricting a study to participants with particular baseline characteristics can reduce variation in potential confounders or make treatment conditions easier to implement consistently.
For example, limiting a nonrandomized comparison to students within a defined prior-achievement range may reduce confounding associated with baseline proficiency.
However, restriction does not automatically eliminate unmeasured confounding or establish exchangeability between comparison groups.
In randomized trials, broad baseline eligibility does not inherently undermine randomization's protection against systematic baseline confounding in expectation.
Excluding heterogeneous participants may simplify implementation or improve precision in some circumstances, but these advantages should not be confused with guaranteed internal validity.
Researchers should evaluate whether a restriction genuinely addresses a bias mechanism or merely creates a more homogeneous sample.
How Can Restrictions Reduce External Validity?
External validity concerns whether a study's findings support inferences beyond the participants and circumstances directly observed.
Restrictions can limit external validity when excluded characteristics influence the outcome, modify intervention effects, or affect whether the intervention can be implemented as intended.
Consider an online learning platform that requires frequent access to high-bandwidth content. Students with limited connectivity may receive a different effective dose of the intervention than students with reliable access.
If the study excludes the former group, its estimated effect may not represent the effect of adopting the platform across an institution with unequal connectivity.
However, not every eligibility difference matters equally. An exclusion involving a characteristic unrelated to the intervention effect may have little consequence for a particular causal generalization.
The central question is whether the excluded characteristic is relevant to the quantity being generalized.
Why Do Effect Modifiers Matter More Than Superficial Similarity?
An effect modifier is a characteristic across which an intervention's causal effect differs on the specified effect scale.
For example, prior digital proficiency may modify the effect of an adaptive learning platform if students with different proficiency levels benefit differently from its features.
Excluding participants with limited digital proficiency could therefore alter the average effect estimated in the study relative to the effect expected in a broader student population.
By contrast, a demographic characteristic may differ between the study and target populations without materially modifying the intervention effect.
Researchers should avoid assuming that every difference creates a generalizability problem of equal importance.
Determining whether a sample adequately represents the target population requires attention to the specific inference, including relevant effect modifiers and the sampling process.
Which Eligibility Restrictions Deserve the Most Scrutiny?
| Restriction |
Possible justification |
Potential applicability concern |
| Minimum prior knowledge |
The intervention requires prerequisite skills. |
Effects may differ among learners who lack those skills. |
| Reliable device and internet access |
Ensures the intervention can be delivered as designed. |
Findings may not apply to resource-constrained environments. |
| Limited comorbidities or complex needs |
May address safety or specific scientific questions. |
Real-world users with multiple needs may be underrepresented. |
| High expected adherence |
Supports consistent exposure during evaluation. |
Implementation effectiveness may differ in ordinary practice. |
| Single institution or language |
Supports feasible administration and consistent measurement. |
Other institutional or linguistic contexts may remain untested. |
| Narrow age or developmental range |
Targets a specific developmental question. |
Findings may not apply to other developmental stages. |
These examples do not imply that researchers should eliminate all restrictions. They illustrate why eligibility decisions should be evaluated against the intended population and intervention.
What Is the Difference Between Excluding Ineligible Participants and Selective Recruitment?
Eligibility criteria establish who qualifies for participation. Recruitment determines which eligible individuals are approached, invited, and enrolled.
A study may have broad eligibility criteria but recruit almost exclusively from convenient institutions. Conversely, a study may recruit systematically from a clearly defined but narrow eligible population.
Both processes influence the final sample, but they operate differently.
Researchers should document the number of individuals assessed for eligibility, reasons for exclusion where appropriate, and participation among those eligible.
Without this information, readers may be unable to determine whether the final sample is narrow because of the study's scientific criteria or because of recruitment and participation processes.
Can Strict Eligibility Criteria Create Selection Bias?
Potentially, but restricted generalizability and selection bias should not be treated as interchangeable.
A study can estimate a valid effect for a narrowly defined eligible population while providing limited evidence about excluded populations.
Selection bias in causal estimation may arise when conditioning on eligibility or participation creates a noncausal association or otherwise distorts the comparison of interest.
For example, selecting participants based on a variable influenced by both exposure and outcome-related causes can introduce collider-related bias under certain causal structures.
However, ordinary restrictions on genuinely preintervention characteristics do not automatically introduce such bias.
Researchers should specify the selection mechanism and causal estimand before concluding that an eligibility criterion creates internal selection bias.
How Do Exclusions Affect the Intervention Being Evaluated?
Eligibility criteria can influence more than the composition of the sample. They may also shape the intervention conditions.
For example, a digital learning program evaluated only among students with reliable connectivity may operate smoothly because participants can access every component.
In a broader population, the same platform might require offline materials, additional technical support, or alternative delivery arrangements.
These modifications could change the intervention strategy itself.
Researchers should therefore distinguish generalizing the effect of the same intervention under comparable conditions from predicting the effect of a modified implementation strategy.
Evidence about one intervention configuration does not automatically establish the effectiveness of every adaptation.
Can Pragmatic Research Designs Improve Real-World Relevance?
Pragmatic trials and evaluations seek to inform decisions under conditions resembling ordinary practice.
They may use broader eligibility criteria, routine service settings, flexible intervention delivery, and outcomes relevant to everyday decision-making.
The PRECIS-2 framework provides a structured way to examine how pragmatic or explanatory a trial is across design domains, including eligibility, recruitment, setting, organization, flexibility, follow-up, outcomes, and analysis.
A more pragmatic design is not automatically superior. Explanatory studies may be appropriate when researchers need to investigate whether an intervention can work under specified conditions.
Pragmatic studies may be more relevant when the question concerns what happens if an institution adopts an intervention in ordinary practice.
The design should follow the decision being investigated rather than an assumption that broader eligibility is always better.
Can Researchers Generalize Findings to Excluded Groups Using Statistical Adjustment?
Sometimes, but only under suitable assumptions and with adequate supporting data.
Generalization or transportability methods may adjust for differences between study participants and a target population when relevant effect modifiers are measured and sufficient overlap exists.
Suppose a study includes students with moderate and high digital proficiency, while the target population contains different proportions of these groups.
With suitable subgroup evidence and population data, researchers may estimate a target-population average effect under defensible assumptions.
However, if the study excludes every student with low digital proficiency, it provides no direct treatment-effect evidence for that subgroup.
Weighting cannot create observations for an entirely absent group.
Extrapolation would require additional assumptions, external evidence, or new data.
The distinction between generalization and transportability becomes especially important when target populations extend beyond the conditions represented in the original study.
Should Researchers Always Broaden Eligibility Criteria?
No. Broad eligibility may improve applicability but can introduce practical and scientific challenges.
Some participants may require intervention adaptations, different outcome measures, or additional support. Broadening eligibility may also increase heterogeneity and complicate interpretation when treatment effects vary substantially.
In some circumstances, the intervention may be inappropriate or unsafe for particular individuals.
The appropriate decision depends on whether the research aims to establish an effect under controlled conditions, evaluate implementation in routine practice, or investigate variation across populations.
Researchers should retain restrictions that are necessary and defensible while questioning exclusions introduced solely for convenience.
How Can Researchers Balance Scientific Rigor and Real-World Applicability?
Begin by identifying the decision the research is intended to inform.
If the goal is to determine whether a specific instructional mechanism can produce learning gains under carefully specified conditions, a focused population may be justified.
If the goal is to inform institution-wide adoption, excluding common categories of intended users may substantially reduce the usefulness of the results.
Researchers can consider stratified recruitment, planned subgroup analyses, implementation monitoring, or complementary studies across different populations.
Subgroup analyses should be planned with attention to statistical power and multiplicity. Failure to detect an interaction does not establish identical effects across groups.
In some research programs, an initial explanatory study followed by broader pragmatic evaluation may provide more informative evidence than expecting one study to answer every question.
How Should Eligibility Criteria Be Reported?
Research reports should describe inclusion and exclusion criteria clearly enough for readers to understand who could participate.
Where relevant, explain why important restrictions were imposed and report how many individuals were excluded for each reason.
Describe the characteristics of enrolled participants and the settings in which the intervention was implemented.
For randomized trials, the CONSORT reporting framework emphasizes transparent reporting of eligibility, recruitment, participant flow, and relevant study characteristics.
Researchers should also identify which excluded populations may differ in ways relevant to intervention effects or implementation.
This information helps others assess whether findings apply to different populations and settings without assuming that every study must be universally representative.
