Can Too Few Eligibility Criteria Make a Research Sample Too Heterogeneous?

Can Including Almost Everyone Make Your Research Question Difficult to Answer?

Suppose you want to investigate whether an AI-supported writing intervention improves academic writing performance. To make the study inclusive, you recruit undergraduate students, postgraduate students, professional researchers, and secondary school learners.

All participants engage in writing, but their tasks, expectations, prior knowledge, and assessment standards differ substantially.

Would one writing performance score mean the same thing across these groups? Could one average intervention effect provide a meaningful answer?

Now consider a clinical study that includes patients with widely different diagnoses, disease stages, and treatment indications under the general category of patients with chronic illness.

Broad eligibility may increase the diversity of participants, but it can also combine people for whom the exposure, intervention, or outcome has different meanings.

How do researchers determine when broad inclusion improves relevance and when insufficient eligibility criteria create heterogeneity that undermines the intended inference?

Yes, When Broad Eligibility Makes the Estimand or Measurement Incoherent

How to Distinguish Useful Population Diversity From Problematic Heterogeneity

What Is Sample Heterogeneity?

Sample heterogeneity refers to variation among participants or other study units in characteristics relevant to the research.

These characteristics may include age, educational level, prior knowledge, clinical severity, institutional context, exposure history, or other attributes.

For example, undergraduate students may differ substantially in academic achievement, digital literacy, and prior experience with generative AI.

Such differences do not automatically make the sample unsuitable.

Heterogeneity becomes a methodological concern when it interferes with the intended measurement, comparison, or interpretation.

Researchers should therefore distinguish the presence of variation from the consequences of that variation for the research question.

What Does It Mean to Have Too Few Eligibility Criteria?

Insufficient eligibility criteria fail to establish the characteristics necessary for meaningful membership in the study population.

For example, defining participants simply as learners may be inadequate for an intervention specifically designed for undergraduate academic writing.

The category could include individuals undertaking substantially different educational tasks.

However, the problem is not the numerical count of criteria.

One carefully specified criterion may be sufficient for a particular research question, while several vague criteria may still leave the population poorly defined.

The adequacy of eligibility depends on whether the rules identify the intended population consistently and meaningfully.

When Is Heterogeneity Scientifically Valuable?

Heterogeneity is valuable when variation across individuals or contexts is part of the phenomenon being studied.

For example, a national study of undergraduate AI literacy may need students from different disciplines, institutions, and socioeconomic circumstances.

Restricting participation to highly proficient technology users would remove variation central to the research question.

Similarly, a pragmatic intervention evaluation may seek to understand effectiveness under ordinary conditions involving participants with diverse characteristics.

Broad eligibility may therefore strengthen relevance when the research design can accommodate and interpret the variation.

When Does Heterogeneity Become Problematic?

Heterogeneity becomes problematic when it creates uncertainty about what is being estimated or whether observations are comparable.

For example, combining secondary school essay scores with doctoral dissertation assessments may produce an outcome lacking a common interpretation.

Similarly, evaluating a treatment across patients for whom the intervention has different clinical indications may create an unclear treatment contrast.

Researchers should examine whether the construct, exposure, intervention, and outcome remain meaningful across included groups.

If not, eligibility may need to be refined or separate estimands established.

Which Types of Heterogeneity Matter?

These dimensions may overlap. The appropriate response depends on which differences are substantively important and whether they can be addressed through measurement, design, or analysis.

How Does Heterogeneity Affect the Estimand?

An estimand is the specific quantity a study intends to estimate.

For example, a causal study may seek the average effect of an AI tutoring intervention among eligible undergraduate students.

If the study also includes postgraduate researchers and secondary school learners, the average effect now concerns a different population.

The resulting estimate may still be mathematically definable, but its substantive meaning depends on the composition of the combined population.

Researchers should specify whether the intended estimand concerns a broad average, a particular subgroup, or variation in effects across groups.

Eligibility criteria should support that estimand rather than allow the population to expand without a clear scientific rationale.

Can a Broad Population Have a Meaningful Average Effect?

Yes. A heterogeneous population can have a well-defined average treatment effect when the intervention, outcome, and target population are specified appropriately.

For example, a study may intentionally estimate the average effect of an instructional intervention across different undergraduate disciplines.

Effects may vary between disciplines, but the population average can remain meaningful for a policy decision concerning all eligible students.

However, the average may conceal important differences in benefit or harm.

Researchers should consider whether subgroup effects or the distribution of effects are also relevant.

Heterogeneity does not invalidate an average merely because individual effects differ.

How Can Heterogeneity Affect Measurement Validity?

Measurement validity concerns whether observations or scores support the intended interpretation.

For example, a writing assessment designed for undergraduate essays may not validly measure the same construct among doctoral researchers completing dissertation chapters.

Even if the same numerical scale is used, the meaning of scores may differ.

Researchers should assess whether instruments, tasks, and administration procedures are appropriate across the included population.

Where necessary, measurement invariance analysis, instrument adaptation, or separate outcome definitions may be considered.

Restricting eligibility is one possible solution, but it is not always preferable to improving measurement.

What Is Measurement Invariance?

Measurement invariance concerns whether a measurement model operates comparably across relevant groups or conditions.

For example, a scale intended to measure AI literacy may function differently across disciplines or educational levels.

If items have different meanings or relationships with the underlying construct, comparisons of observed scores may be misleading.

Measurement invariance can be evaluated using appropriate psychometric methods, depending on the instrument and study design.

However, not every outcome requires the same type of invariance analysis.

Researchers should choose comparability assessments appropriate to the construct, measurement model, and intended comparison.

How Does Heterogeneity Affect Statistical Precision?

Greater outcome variability can reduce precision for some estimates when sample size and other design features remain constant.

For example, a study combining participants with substantially different baseline writing abilities may have greater variability in writing scores than a study restricted to one proficiency level.

However, restriction can also reduce the eligible sample size and alter the expected intervention effect.

Statistical precision therefore depends on more than population homogeneity.

Appropriate covariate adjustment, stratification, or design features may improve precision while preserving relevant population variation.

Researchers should not assume that narrowing eligibility is always the most efficient solution.

Does Heterogeneity Automatically Reduce Statistical Power?

No. The relationship depends on the statistical test, outcome variability, effect distribution, sample size, and design.

Greater unexplained outcome variance may reduce power for certain comparisons.

However, including a broader population may increase recruitment and provide more information about relevant subgroups.

Heterogeneity may also be central to the hypothesis, such as when researchers test whether an intervention effect differs by prior achievement.

Power calculations should reflect the intended estimand and analytical strategy rather than a general assumption that heterogeneous samples are inferior.

Can Heterogeneity Introduce Confounding?

Population heterogeneity does not itself constitute confounding.

Confounding depends on the causal relationships among exposure, outcome, and other variables.

For example, students may differ in prior achievement, but this variation creates confounding only under relevant conditions involving exposure assignment and outcomes.

Restricting the population to students with similar achievement may reduce some confounding in an observational study, but it may not eliminate other sources.

Researchers should distinguish heterogeneity from confounding and avoid using eligibility restrictions as a universal substitute for causal design.

What Is Effect Heterogeneity?

Effect heterogeneity occurs when the effect of an intervention or exposure differs across individuals or subgroups.

For example, an AI tutoring intervention may produce different effects among students with high and low prior knowledge.

These differences may be scientifically important, particularly when the intervention is intended for broad implementation.

Restricting eligibility to one subgroup would prevent direct estimation of effects in excluded groups.

Researchers should consider whether the study aims to estimate an overall average effect, subgroup effects, or both.

Relevant subgroup analyses should be planned with attention to adequate sample size and multiplicity.

Should Researchers Restrict Eligibility to Improve Homogeneity?

Only when the restriction has a defensible scientific or methodological purpose.

For example, limiting an intervention to students enrolled in courses involving comparable writing tasks may improve conceptual and measurement alignment.

However, restricting further to students with high grades or extensive AI experience may remove important variation without improving the central comparison.

Researchers should ask whether the characteristic is necessary for eligibility or could instead be measured and addressed analytically.

Unnecessary restriction may reduce generalizability and limit the study's relevance to ordinary practice.

Can Stratification Address Heterogeneity Without Excluding Participants?

Yes. Stratification can help ensure representation or permit analysis across relevant population subgroups.

For example, a study of AI literacy might stratify students by discipline or institutional type.

In randomized studies, stratified randomization may help balance important baseline characteristics across treatment groups.

However, stratification requires appropriate planning and analysis.

Too many strata relative to the available sample may create sparse categories and operational difficulties.

Researchers should select stratification variables based on substantive relevance rather than attempting to stratify every observed difference.

Can Multilevel Models Address Population Heterogeneity?

Multilevel models may be useful when observations are nested within groups such as classrooms, schools, hospitals, or institutions.

For example, students within the same university may share instructional policies or digital infrastructure.

A multilevel model can represent relevant between-group and within-group variation under appropriate assumptions.

However, multilevel modeling does not automatically resolve conceptual incompatibility or measurement noncomparability.

Nor does it guarantee that a conveniently selected set of institutions represents a broader population.

Statistical modeling should complement, not replace, a coherent population definition.

When Should Researchers Use Separate Subgroup Analyses?

Subgroup analyses may be appropriate when the research question concerns meaningful differences between categories of participants.

For example, an educational intervention may be evaluated separately among novice and experienced users.

However, subgroup estimates can be imprecise when sample sizes are small.

Researchers should distinguish prespecified subgroup analyses from exploratory comparisons developed after observing the results.

Where interaction effects are central, the study should be designed with sufficient information to estimate them.

Separate analyses should not be used merely to avoid explaining an otherwise poorly defined combined population.

Can Broad Eligibility Improve Generalizability?

Broad eligibility may increase the range of people or settings directly represented by a study.

For example, including students with different levels of achievement and digital access may make an educational intervention evaluation more relevant to routine implementation.

However, broader eligibility does not automatically guarantee representative sampling.

Coverage, recruitment, nonresponse, and the analytical design still determine the strength of population inference.

Researchers should distinguish broad eligibility from actual representation of the intended target population.

How Does the Explanatory-Pragmatic Distinction Apply?

Explanatory studies may use more focused eligibility to investigate effects under specified conditions.

Pragmatic studies may use broader eligibility to evaluate effectiveness under routine practice conditions.

However, these approaches differ along several design dimensions beyond eligibility.

The PRECIS-2 framework considers eligibility alongside recruitment, setting, intervention delivery, follow-up, outcomes, and analysis.

Researchers should choose eligibility breadth according to the intended purpose rather than assuming that pragmatic research is always better or that explanatory research requires maximal restriction.

What If the Population Includes Participants With Different Diagnoses?

Clinical heterogeneity may be appropriate when the intervention targets a common mechanism or service need across diagnoses.

For example, a rehabilitation intervention might be relevant to patients with different conditions but similar functional limitations.

However, combining diagnoses may be inappropriate if treatment indications, outcome definitions, or expected effects differ substantially.

Researchers should determine whether a common estimand remains meaningful.

Where necessary, diagnosis-specific strata, subgroup analyses, or separate studies may be preferable.

Clinical diversity should be justified by the intervention and research question rather than included or excluded automatically.

What If Participants Have Different Levels of Experience?

Experience heterogeneity can be valuable when the research concerns adoption, learning, or variation in practice.

For example, a study of AI literacy may appropriately include novice and experienced users.

However, a qualitative study investigating advanced AI-assisted systematic review practices may require participants with relevant firsthand experience.

The researcher should determine whether experience defines the phenomenon or represents variation within the target population.

Experience should be operationalized using relevant activities, frequency, duration, or demonstrated competence rather than vague labels.

How Do You Know Whether Eligibility Is Too Broad?

Begin by examining whether all eligible participants can meaningfully contribute to the intended outcome or comparison.

Ask whether the intervention is appropriate across included groups and whether the outcome measures the same construct.

Then consider whether important differences can be addressed through stratification, measurement adaptation, or appropriate analysis.

If the combined population lacks a coherent estimand, eligibility may need refinement.

If heterogeneity is relevant to the question and can be analyzed meaningfully, broad inclusion may be preferable.

The decision should follow scientific reasoning rather than a desire for uniform participants.

How Should Researchers Revise Insufficient Eligibility Criteria?

Identify the minimum characteristics necessary for population membership, intervention applicability, and valid measurement.

For example, an AI-supported writing intervention may require participation in courses involving comparable academic writing tasks.

It may not require identical academic achievement, device ownership, or prior AI experience.

Researchers should distinguish essential eligibility conditions from characteristics that can be measured and analyzed.

Where the research question genuinely spans different populations, separate estimands or subgroup-specific measures may be more appropriate than simply excluding participants.

Revisions should be documented and completed before outcome examination where feasible.

How Should Population Heterogeneity Be Reported?

Methods sections should describe the target population, eligibility criteria, and characteristics relevant to heterogeneity.

Explain why the included groups can meaningfully contribute to the intended estimand.

Describe any stratification, subgroup analysis, measurement comparability assessment, or multilevel modeling used.

Where relevant, report subgroup sizes and the uncertainty associated with subgroup estimates.

In the discussion, identify whether the findings represent an overall population average, subgroup-specific relationships, or context-dependent results.

Transparent reporting helps readers determine whether heterogeneity was appropriately incorporated into the research rather than ignored.

When an AI Writing Intervention Includes Participants With Incomparable Writing Tasks

Suppose the researcher calculates one overall average improvement of six points across the four groups.

Without a common, interpretable measurement scale, the numerical average may not represent a meaningful improvement in academic writing performance.

However, the problem is not simply that the participants are diverse.

If the researcher develops a defensible common outcome measure and defines a meaningful population-wide estimand, the broader study may be appropriate.

Alternatively, separate evaluations may provide clearer evidence about the intervention's effectiveness in different educational contexts.

The example illustrates that eligibility should establish conceptual coherence while preserving the variation necessary to answer the intended question.

Common Misconceptions About Insufficient Eligibility Criteria and Heterogeneity

How to Decide Whether Your Eligibility Criteria Are Too Broad

Begin by identifying whether all eligible participants contribute to a coherent research question and estimand.

Then evaluate whether important differences should lead to refined eligibility, improved measurement, or explicit subgroup analysis.

The objective is not to eliminate heterogeneity. It is to ensure that variation is either scientifically meaningful, appropriately measured, or addressed through a design capable of supporting the intended inference.

What to Verify Before Finalizing Broad Eligibility Criteria

Frequently Asked Questions About Eligibility Criteria and Sample Heterogeneity

Include Meaningful Diversity Without Losing Conceptual Coherence

How to Cite This Guide

This guide is intended to be read, shared, and used in research, teaching, and academic work. If you draw on its ideas, explanations, or other content, please acknowledge the source by citing the guide. Doing so gives appropriate credit and helps your readers locate the original resource.