03 · What You Need to Know
When Eligibility Restrictions Protect Research Design and When They Distort It
What Are Eligibility Criteria?
Eligibility criteria are rules determining which individuals or units may participate in a study or enter a specified analysis.
Inclusion criteria identify required characteristics, while exclusion criteria identify circumstances that prevent participation.
For example, a study may include first-year engineering students enrolled in a particular course and exclude students who have already completed equivalent instruction.
Eligibility criteria can support ethical recruitment, ensure that the intervention is appropriate, and define the population for which the research question is meaningful.
They may also improve comparability or reduce irrelevant variation.
However, their methodological consequences depend on the variables used and the stage at which eligibility is determined.
Why Are Eligibility Criteria Not Automatically Selection Bias?
Every empirical study has population boundaries. Restricting eligibility does not necessarily distort an estimate.
Suppose a randomized experiment includes only students enrolled in introductory programming courses. If eligibility is established before treatment assignment, randomization can support a credible causal comparison among those eligible students.
The resulting effect may not apply to postgraduate students or students in other disciplines, but this is primarily a question of external validity.
Selection bias becomes a concern when the eligibility process changes the relationship being estimated in a way inconsistent with the target estimand.
Researchers should therefore distinguish internal validity from external validity when evaluating restrictions.
How Can Eligibility Criteria Introduce Selection Bias?
Eligibility criteria can create bias when the variable determining inclusion is related to exposure and outcome through a causal structure that makes conditioning problematic.
For example, suppose a study investigates the association between use of an educational technology platform and academic achievement.
Researchers include only students who completed the course.
If course completion is influenced by technology use and by other factors related to achievement, conditioning on completion may create or distort associations.
The precise consequences depend on the underlying causal relationships.
Researchers should not infer selection bias merely because participants were excluded. They must identify how the restriction affects the intended comparison.
Why Does the Timing of Eligibility Matter?
Eligibility determined before exposure or treatment assignment often has different implications from eligibility determined afterward.
For example, restricting a randomized trial to students with a specified baseline academic level can define a narrower eligible population while preserving the randomized comparison within that population.
By contrast, excluding students who fail to complete the intervention after randomization can undermine the original treatment-group comparability.
Completion may be affected by treatment assignment, motivation, academic difficulties, or other outcome-related factors.
Researchers should therefore distinguish baseline eligibility from postassignment analytical exclusions.
Calling both procedures eligibility criteria does not make their causal consequences equivalent.
What Is Collider Stratification Bias?
A collider is a variable influenced by two or more other variables in a causal structure.
Conditioning on a collider can create an association between its causes even when those causes were independent before conditioning.
For example, suppose course completion is influenced by both participation in an intervention and students' prior academic preparation.
Among students who complete the course, intervention assignment and academic preparation may become associated because both influence the probability of being included.
If academic preparation also affects examination performance, the selected treatment groups may differ in outcome-related characteristics.
This can distort the treatment comparison.
Researchers should examine how conditioning on study participation or selection variables can create misleading associations.
Can Restricting Eligibility Improve Internal Validity?
Yes, under appropriate circumstances.
Restriction may reduce confounding in observational research by limiting variation in a measured confounder.
For example, researchers may restrict an observational comparison to students within one academic level to reduce differences associated with progression through a program.
However, restriction does not automatically eliminate confounding from other variables.
It can also reduce sample size, narrow the target population, and sometimes introduce selection bias depending on the restriction variable's causal role.
Researchers should evaluate the trade-off between restricting confounding and limiting generalizability rather than treating restriction as universally beneficial.
How Can Eligibility Criteria Limit Generalizability?
Eligibility restrictions may exclude groups whose characteristics differ from the intended users of an intervention.
For example, a trial evaluating an online learning platform may require reliable personal internet access.
The intervention may be effective among eligible students, but the study provides limited direct evidence about learners with intermittent connectivity.
If connectivity modifies the treatment effect, the average effect in the broader population may differ.
This is an external validity concern even when the within-study randomized comparison is internally valid.
Researchers should consider whether strict eligibility criteria undermine real-world applicability before generalizing beyond the eligible population.
Can Eligibility Based on Prior Outcomes Create Bias?
Potentially. Selecting participants according to previous outcomes may alter the distribution of factors associated with those outcomes.
For example, recruiting only students with exceptionally low examination scores can create a sample in which subsequent improvement partly reflects regression to the mean.
Regression to the mean is not identical to selection bias, but selection based on extreme measurements can complicate causal interpretation.
Similarly, restricting a study to students who previously succeeded may exclude individuals with different combinations of preparation, resources, and support.
The consequences depend on whether the restriction defines the intended population or introduces a problematic selection mechanism.
Researchers should distinguish regression to the mean from other threats to internal validity.
Can Eligibility Depend on Variables Affected by Exposure?
Yes, particularly in observational or retrospective studies.
Suppose researchers investigate whether prior use of a learning platform influences academic performance but include only students who remained enrolled after the platform was introduced.
Continued enrollment may have been influenced by platform use and other academic circumstances.
Restricting the analysis to remaining students can condition on a postexposure variable.
Such conditioning may distort the association, depending on the causal structure.
Researchers should identify when eligibility was assessed relative to exposure and outcome development.
Retrospective eligibility rules deserve particular scrutiny because the available records may already reflect consequences of the exposure being investigated.
What Is the Difference Between Eligibility Restrictions and Loss to Follow-Up?
Eligibility restrictions determine who may enter the study or analysis under specified rules.
Loss to follow-up occurs when enrolled participants subsequently become unavailable for outcome assessment.
These processes can overlap when researchers retrospectively exclude participants who lack follow-up information.
For example, a protocol may initially enroll all eligible students but later analyze only those who complete the final examination.
That decision should not be presented as though the excluded students were never eligible.
Researchers should distinguish initial eligibility, recruitment, retention, and analytical inclusion.
The consequences of loss to follow-up for selection bias require separate evaluation.
Can Complete-Case Eligibility Rules Create Bias?
Yes. Requiring complete data for every variable can create a selected analytical sample.
Suppose students with poor connectivity are more likely to have missing platform records and lower examination performance.
Excluding incomplete records may disproportionately remove those students.
The resulting estimate may differ from the intended population estimate.
However, the direction and magnitude of bias depend on the missingness mechanism and estimand.
Researchers should investigate missing-data patterns before converting complete-record availability into an eligibility rule.
Can Matching or Restriction Create Similar Selection Problems?
Yes. Matching and restriction both determine which observations contribute to an analysis.
Matching may improve comparability on selected variables, but inappropriate matching variables or selective availability of matches can create additional concerns.
Restriction may reduce variation in a confounder while narrowing the study population.
Neither method should be evaluated solely by whether the resulting groups appear more similar.
Researchers should examine the causal role of the variables used and the population represented by the resulting sample.
The implications of matching for confounding and selection bias depend on the design and implementation.
Should Eligibility Criteria Be Broader Whenever Possible?
Not necessarily. Broader eligibility may improve relevance to routine practice, but it can also introduce ethical, clinical, logistical, or measurement challenges.
For example, an intervention requiring a particular prerequisite skill may not be appropriate for participants who have not acquired that skill.
Similarly, including participants for whom the intervention cannot be delivered safely or meaningfully may not improve the study.
The appropriate criteria should reflect the research question, intervention requirements, participant protection, and intended population.
The objective is not maximum inclusiveness at any cost. It is justified inclusion without unnecessary restrictions that undermine the intended inference.
How Can Researchers Evaluate Eligibility Criteria Using a Causal Diagram?
A directed acyclic graph can represent hypothesized relationships among exposure, outcome, eligibility, and relevant common causes.
Researchers can examine whether eligibility is a baseline characteristic, a cause of exposure, a consequence of exposure, or a common effect of variables related to the outcome.
For example, a diagram may show that intervention assignment and academic preparation both influence course completion.
Conditioning on completion may then open a noncausal path between assignment and academic preparation.
The diagram does not establish that the assumed relationships are true, but it helps make the selection assumptions explicit.
Researchers should use substantive knowledge to develop the diagram rather than selecting variables solely from observed statistical associations.
Can Statistical Adjustment Repair Bias Introduced by Eligibility?
Sometimes, but not automatically.
Appropriate methods may include weighting, standardization, or other approaches designed for the relevant selection mechanism.
However, their validity depends on measured variables, sufficient overlap, and assumptions about selection and outcomes.
When important eligible groups are entirely absent, statistical models may require extrapolation beyond the observed evidence.
Unmeasured causes of selection and outcomes may also prevent identification.
Researchers should prioritize defensible eligibility rules during design rather than assuming that later adjustment can correct every restriction.
How Should Eligibility Criteria Be Justified in a Protocol?
Each criterion should have a clear methodological, ethical, or practical rationale.
Researchers should specify whether the criterion defines the target population, protects participants, ensures intervention feasibility, reduces confounding, or supports measurement.
They should identify when eligibility is assessed and whether the criterion may be affected by exposure or treatment assignment.
Where broader generalization is intended, explain which groups are excluded and whether those exclusions could modify the effect.
Eligibility criteria should be prespecified whenever feasible and changes should be documented.
Transparent justification helps distinguish necessary design boundaries from restrictions that may distort the intended comparison.
