Should Every Potential Confounder Become an Exclusion Criterion?

Should You Exclude Participants Who Have Characteristics That Might Affect Your Results?

Suppose you are investigating whether generative AI use is associated with academic writing performance among undergraduate students.

You recognize that prior writing ability, academic motivation, digital literacy, socioeconomic circumstances, and access to instructional support may influence both students' use of AI and their writing performance.

Would the study become more valid if you excluded students with low prior achievement, limited digital literacy, additional tutoring, or other characteristics that might complicate the relationship?

At first, restriction appears attractive. If everyone is similar, perhaps there will be fewer alternative explanations for the findings.

However, excluding participants can also change the population being studied, remove meaningful variation, and introduce selection problems. Moreover, not every characteristic associated with an outcome is necessarily a confounder.

How should researchers decide when restriction is appropriate and when potential confounders should instead be measured or addressed through the research design and analysis?

No. Confounding Should Be Addressed According to the Causal Question, Not Through Automatic Exclusion

When Restriction Helps Control Confounding and When It Creates New Problems

What Is a Confounder?

A confounder is a variable that contributes to distortion of the causal relationship between an exposure and an outcome under the relevant causal structure.

In a conventional observational setting, a confounder may influence both exposure assignment and the outcome.

For example, students' prior academic achievement may influence their likelihood of using an AI tutoring system and their subsequent academic performance.

If prior achievement is not appropriately addressed, the observed association between AI use and performance may partly reflect differences that existed before AI use.

However, a variable is not a confounder merely because it predicts the outcome or is statistically associated with the exposure.

Confounding must be evaluated in relation to the intended causal effect and the underlying relationships among variables.

Why Is Excluding Every Potential Confounder Problematic?

Exclusion removes individuals who possess particular characteristics, thereby changing the eligible population.

For example, excluding students with low academic achievement may reduce variation in baseline performance but also eliminate students who could benefit substantially from an educational intervention.

Excluding students with limited internet access may remove a source of variation while restricting the study to comparatively well-connected participants.

Such restrictions may reduce the relevance of findings to ordinary educational settings.

Moreover, participants may still differ in other confounding characteristics.

Restriction therefore does not guarantee exchangeability between exposure groups or eliminate the need for a defensible causal design.

What Is Restriction as a Confounding-Control Method?

Restriction is a design-stage approach that limits eligibility to individuals within specified categories or ranges of a variable.

For example, a study might include only students enrolled in the same academic year to reduce variation associated with educational level.

When the restricted variable is a relevant confounder, limiting its variation may reduce or eliminate confounding attributable to that variable within the restricted population.

However, restriction may not control other confounders, measurement error, or residual variation within the selected category.

It also changes the population for which the effect is estimated.

The method should therefore be selected deliberately rather than used as a general response to every variable that might influence results.

How Does Restriction Compare With Other Confounding-Control Methods?

No method is universally superior. Researchers should select an approach that supports the intended causal estimand while preserving the population and comparisons necessary for the question.

When Might Restriction Be Appropriate?

Restriction may be appropriate when a characteristic is strongly relevant to the research question, can be assessed reliably, and defines a scientifically meaningful subgroup.

For example, a study of a curriculum-specific instructional intervention may reasonably focus on students enrolled in courses with comparable learning objectives.

Restriction may also be useful when an important confounding variable has a small number of clearly defined categories and broader inclusion would introduce substantial complications.

However, researchers should determine whether the restriction is necessary for intervention applicability, measurement comparability, or confounding control.

These are related but distinct justifications.

A restriction that improves one aspect of the study may simultaneously reduce the population to which findings apply.

What Is the Difference Between a Confounder and a Prognostic Variable?

A prognostic variable predicts the outcome, but it does not necessarily confound the exposure-outcome relationship.

For example, baseline writing ability may strongly predict subsequent writing performance.

In a properly randomized trial, baseline writing ability does not systematically determine treatment assignment through the randomization mechanism, although chance imbalances may occur.

Adjusting for baseline achievement may improve precision, but excluding low-achieving students is not automatically necessary for confounding control.

Researchers should distinguish prognostic relevance from the causal role a variable plays in exposure assignment and outcome generation.

What Is the Difference Between a Confounder and an Effect Modifier?

An effect modifier is a characteristic across whose levels the effect of an exposure or intervention differs.

For example, an AI tutoring intervention might have different effects among students with high and low baseline achievement.

If prior achievement modifies the intervention effect, restricting the population to high-achieving students prevents direct estimation of the effect among low-achieving students.

A characteristic can have several causal roles, but confounding and effect modification are conceptually distinct.

Researchers should consider whether important population variation should be preserved for subgroup or interaction analysis rather than eliminated through eligibility restrictions.

Why Does Timing Matter When Identifying Confounders?

Confounding control generally requires attention to variables that precede the exposure or treatment decision in the relevant causal structure.

For example, prior academic achievement measured before AI tutoring begins may be relevant to treatment selection and subsequent performance.

However, achievement measured after tutoring begins may already have been influenced by the intervention.

Conditioning on such a post-treatment variable can change the causal estimand or introduce bias.

Researchers should therefore identify when each variable is measured and whether it may be affected by the exposure.

Temporal ordering is essential for distinguishing baseline confounding from mediation and post-exposure selection.

Why Should Mediators Not Automatically Become Exclusion Criteria?

A mediator lies on a causal pathway through which an exposure affects an outcome.

For example, an AI tutoring intervention may improve academic performance partly by increasing practice time.

If practice time is affected by the intervention, excluding students based on their subsequent practice time may remove part of the intervention process and introduce selection problems.

Similarly, adjusting for a mediator when estimating a total effect can alter the quantity being estimated.

Researchers should distinguish confounders from mediators using substantive knowledge and causal reasoning rather than relying solely on statistical associations.

What Is Collider Bias, and How Can Restriction Create It?

A collider is a variable influenced by two or more other variables in a causal structure.

Conditioning on a collider can create a statistical association between its causes even when they would otherwise be independent.

For example, participation in an educational technology study might be influenced by both students' motivation and their prior technology experience.

Restricting the study to highly motivated volunteers or conditioning on participation may create associations among relevant characteristics under certain causal structures.

This can distort exposure-outcome relationships.

Researchers should not assume that restricting a sample to apparently comparable participants always reduces bias.

Sometimes the restriction itself creates a new source of distortion.

How Can Directed Acyclic Graphs Help?

Directed acyclic graphs, or DAGs, represent hypothesized causal relationships among variables.

They can help researchers identify potential confounding pathways, mediators, colliders, and variables that should or should not be conditioned on for a particular causal estimand.

For example, a DAG concerning AI tutoring and achievement might include baseline achievement, motivation, digital access, tutoring use, and subsequent performance.

The graph can clarify whether a proposed eligibility restriction blocks a confounding pathway or creates an unwanted selection condition.

However, a DAG is only as defensible as the causal assumptions used to construct it.

Researchers should develop causal diagrams using substantive knowledge rather than treating them as automatic statistical solutions.

What Is Residual Confounding?

Residual confounding occurs when confounding remains after an attempted control strategy.

For example, restricting participants to one academic year may reduce differences in educational stage but leave variation in prior achievement, socioeconomic circumstances, or motivation.

Similarly, a broadly defined severity category may contain meaningful differences in clinical prognosis.

Measurement error and incomplete classification can also leave residual confounding.

Researchers should therefore avoid claiming that restriction completely eliminates confounding unless the assumptions supporting that claim are defensible.

Can Restriction Create Positivity Problems?

Positivity concerns whether the relevant exposure or treatment alternatives are possible within the covariate patterns required for the intended causal comparison.

For example, if students with certain characteristics never receive a particular educational intervention, estimating its effect for those students may require extrapolation beyond the observed data.

Restriction may sometimes define a population with better treatment overlap, but it also changes the target estimand.

Conversely, unnecessarily narrow eligibility may remove groups important to the original population of interest.

Researchers should distinguish improving overlap within a restricted population from estimating the effect for the original broader population.

How Does Restriction Affect Generalizability?

Exclusion based on potential confounders can produce a population that differs from the intended population of application.

For example, a study of educational AI effectiveness restricted to students with reliable connectivity may not directly represent students with limited access.

If connectivity modifies the intervention effect, the restricted effect may differ from the broader population effect.

However, a narrow population does not automatically produce internally valid estimates, nor does a broad population automatically guarantee generalizability.

The consequences of narrow population definitions for validity and generalizability should be evaluated separately.

Should Researchers Exclude Participants With Comorbidities?

Not automatically. In clinical observational research, comorbidities may influence treatment selection and outcomes.

However, excluding all patients with additional conditions can produce a study population that differs substantially from routine clinical practice.

Researchers should determine whether a comorbidity creates a genuine contraindication, prevents meaningful measurement, or represents a potential confounder that could be measured and addressed.

Restriction may be justified in some circumstances, but the rationale should be explicit.

Clinical complexity alone is not sufficient justification for excluding otherwise relevant patients.

What About Confounding by Indication?

Confounding by indication arises when the clinical reasons for prescribing a treatment are related to the outcome being studied.

For example, patients with more severe disease may be more likely to receive an intensive intervention and also have poorer outcomes independent of treatment.

A crude comparison between treated and untreated patients may therefore be misleading.

Restricting the study to one severity category may reduce some variation but may not eliminate differences in clinical indications.

Researchers may need detailed baseline information, appropriate comparator selection, and causal adjustment methods.

Exclusion should not be treated as a universal remedy for treatment-selection mechanisms.

When Is Randomization Preferable to Restriction?

When feasible and ethically appropriate, randomization can support causal inference by assigning treatment independently of baseline characteristics in expectation.

For example, randomly assigning eligible students to AI-supported tutoring or conventional instruction can reduce systematic baseline confounding associated with treatment selection.

Randomization does not guarantee perfect balance in every realized sample, and other threats such as attrition, nonadherence, and measurement error remain possible.

Nevertheless, it can reduce the need to exclude participants merely to make treatment groups appear comparable.

Eligibility should still reflect intervention relevance, safety, and the intended population.

When Should Confounders Be Measured Instead of Used for Exclusion?

Measurement may be preferable when the characteristic is common, substantively relevant, and can be assessed with sufficient validity.

For example, prior academic achievement, digital literacy, and socioeconomic circumstances may be measured before an educational intervention.

Researchers can then evaluate appropriate adjustment or subgroup analyses.

However, collecting a variable does not guarantee that adjustment will eliminate confounding.

Analytical methods require assumptions concerning measurement, model specification, and the absence of important unmeasured confounding.

Variable selection should be informed by the causal structure rather than automated significance testing alone.

Can Matching Replace Exclusion Criteria?

Matching may construct comparison groups with similar measured characteristics.

For example, students using AI tutoring could be matched with nonusers who have similar baseline achievement and other relevant characteristics.

However, matching can exclude unmatched individuals and thereby change the population represented by the analysis.

It also cannot automatically eliminate unmeasured confounding.

Researchers should identify the estimand targeted by the matching strategy and assess balance, overlap, and remaining uncertainty.

Matching should not be treated as a method that preserves every advantage of broad eligibility without trade-offs.

Can Statistical Adjustment Introduce Bias?

Yes. Adjusting for inappropriate variables can introduce bias or change the intended causal estimand.

For example, controlling for a mediator when estimating a total treatment effect may remove part of the effect of interest.

Conditioning on a collider may open an otherwise blocked noncausal pathway.

Researchers should therefore avoid the assumption that adjusting for every available variable is always beneficial.

The same causal reasoning used to evaluate exclusion criteria should guide statistical covariate selection.

How Should Researchers Decide Between Restriction and Adjustment?

The decision should consider the research question, target population, causal structure, measurement quality, available sample, and intended estimand.

Restriction may be appropriate when a subgroup is scientifically central or when an intervention cannot meaningfully be delivered outside it.

Adjustment may be preferable when the researcher wants to retain relevant population variation and suitable confounder measurements are available.

In some studies, both approaches may be used for different purposes.

For example, researchers may restrict eligibility to students enrolled in relevant programming courses while adjusting for baseline achievement.

The objective is not to eliminate every difference between participants, but to obtain a valid comparison for the intended causal question.

How Can Researchers Audit Proposed Confounder-Based Exclusions?

For every proposed exclusion, identify the variable's hypothesized causal role.

Determine whether it is a confounder, mediator, collider, effect modifier, prognostic variable, or another characteristic.

Then ask whether exclusion is necessary or whether measurement and appropriate analysis would better preserve the target population.

Consider how the restriction affects sample size, treatment overlap, and generalizability.

Finally, document the decision before examining study outcomes.

This process can prevent eligibility criteria from becoming an indiscriminate list of characteristics that happen to correlate with the outcome.

How Should Confounding-Control Decisions Be Reported?

Methods sections should explain which variables were considered potential confounders and why.

Describe restrictions, matching procedures, stratification, weighting, or adjustment methods used to address them.

Specify the timing of covariate measurement and the intended estimand.

Where appropriate, explain the causal assumptions underlying variable selection.

Report relevant limitations, including residual or unmeasured confounding and the consequences of population restriction.

Transparent reporting helps readers evaluate whether the analysis supports the claimed causal interpretation.

Should Students With Low Achievement or Limited Digital Access Be Excluded?

Suppose students using AI tutoring obtain an average score of 84, while nonusers obtain an average score of 78.

The crude six-point difference does not establish that tutoring caused the improvement.

Students choosing to use the platform may already differ in motivation, prior achievement, or other relevant characteristics.

Excluding every student with lower achievement or limited digital access would not automatically solve this problem. It would instead change the population being studied and potentially remove important variation.

A more defensible approach is to preserve relevant eligibility, measure plausible baseline confounders, specify an appropriate causal analysis, and acknowledge remaining assumptions.

If causal identification cannot be adequately supported, the researcher should report the association without presenting it as an established intervention effect.

Common Misconceptions About Confounders and Exclusion Criteria

How to Decide Whether a Potential Confounder Should Lead to Exclusion

Begin by identifying the variable's causal role rather than assuming every characteristic associated with the outcome requires control.

Then choose the design or analytical approach that best supports the intended estimand while preserving relevant population coverage.

The objective is not to eliminate every difference between participants. It is to establish an appropriate comparison for the research question using defensible causal assumptions and methods.

What to Verify Before Excluding Participants Because of Potential Confounding

Frequently Asked Questions About Confounders and Exclusion Criteria

Control Confounding Without Automatically Excluding Relevant Participants

How to Cite This Guide

This guide is intended to be read, shared, and used in research, teaching, and academic work. If you draw on its ideas, explanations, or other content, please acknowledge the source by citing the guide. Doing so gives appropriate credit and helps your readers locate the original resource.