Can Generative AI Apply Systematic Review Inclusion and Exclusion Criteria Reliably?

Can AI Apply Your Eligibility Criteria the Way a Trained Reviewer Would?

Inclusion and exclusion criteria determine which studies contribute to a systematic review. Two articles may investigate similar topics yet differ in study design, participants, intervention characteristics, or outcomes in ways that determine whether they qualify.

Generative AI can compare a research article with written instructions and produce an eligibility recommendation. The difficulty is that review criteria often require interpretation rather than simple keyword matching. A model might recognize that an article concerns university students but overlook that the intervention does not meet the review's definition.

Can researchers rely on AI to apply these criteria consistently, particularly when a mistaken exclusion could remove relevant evidence from the review?

AI Can Apply Criteria, but Reliability Must Be Demonstrated

Why Applying Eligibility Criteria Is More Difficult Than It Appears

What Are Inclusion and Exclusion Criteria?

Inclusion criteria specify the characteristics studies must satisfy to qualify for a review. Exclusion criteria identify characteristics that make studies ineligible. Together, they operationalize the review question and establish the boundaries of the evidence base.

Depending on the review, eligibility may concern participants, interventions or exposures, comparators, outcomes, study designs, settings, time periods, and report characteristics. Frameworks such as PICO can help organize some of these decisions, although not every review question is best expressed through PICO.

Criteria should be established before screening and documented in the review protocol. Researchers should distinguish genuine protocol amendments from clarifications needed to apply existing criteria consistently.

Why Does AI Need More Than a List of Keywords?

Eligibility is a methodological judgment, not simply a test of whether certain words appear in an article.

Consider a review of randomized controlled trials investigating AI-generated formative feedback among university students. An article might contain the terms "random," "artificial intelligence," "feedback," and "students" without meeting the eligibility criteria.

For example, "randomly selected participants" describes sampling, not necessarily random allocation to intervention conditions. Likewise, feedback delivered through an online platform is not necessarily generated by AI.

A model that relies on superficial similarities may classify these studies incorrectly. A more appropriate assessment examines what the researchers actually did.

Which Eligibility Criteria Are Particularly Difficult for AI?

Some criteria can be assessed from clearly reported facts. Others require distinguishing closely related methodological concepts or interpreting information distributed across several sections of a report.

The difficulty of a criterion depends on how it is operationalized and reported. A seemingly simple population criterion can become complicated when a study includes several age groups or educational levels.

How Can Researchers Make Criteria More Operational?

Before applying AI, researchers should define each criterion in terms that support consistent decisions. This may include specifying what evidence confirms eligibility, what clearly establishes ineligibility, and what information would leave the decision unresolved.

For example, the criterion "studies involving AI feedback" is broad. A more operational definition could specify feedback generated through a machine-learning or generative AI system in response to student work, while excluding studies in which a digital platform merely delivers feedback written by instructors.

Researchers should also specify how mixed populations, multicomponent interventions, and studies reporting several outcomes will be handled.

These definitions should reflect the review question and protocol rather than being invented retrospectively to make AI classifications appear more consistent.

Should AI Assess Each Criterion Separately?

Criterion-by-criterion assessment can make the screening process easier to inspect. Instead of requesting a single include-or-exclude answer, researchers can ask AI to identify evidence relevant to each requirement and provide a provisional classification.

A structured output might distinguish between a criterion being met, not met, or insufficiently reported. It could also identify the supporting passage and its location in the source document.

This structure improves traceability but does not guarantee accuracy. The model may extract the correct passage yet misunderstand its implications.

Consequently, researchers should verify both the cited evidence and the conclusion drawn from it.

How Should AI Handle Missing or Ambiguous Information?

A crucial distinction is the difference between a study failing an eligibility criterion and a report not providing enough information to evaluate that criterion.

Suppose an abstract does not mention how participants were allocated. That omission does not establish that randomization was absent. Similarly, a full-text article may omit information available in a trial registration, supplementary document, or related report.

At the title and abstract screening stage, uncertainty will often justify retaining a record for further assessment. During full-text screening, researchers may need to investigate related sources or seek clarification.

AI should not convert missing information into a confident exclusion without an appropriate methodological basis.

How Reliable Is Generative AI at Eligibility Classification?

Published evaluations suggest that large language models can perform some eligibility-related tasks, but results vary considerably across datasets and review procedures.

Khraisha and colleagues (2024) evaluated GPT-4 across screening and extraction activities involving different literature types and languages. Their findings illustrate why performance should be assessed using several measures rather than relying exclusively on overall accuracy.

A systematic review of generative AI applications in evidence synthesis also reported substantial variation in incorrect exclusion decisions across the comparative studies examined. These findings do not establish a universal error rate for all models or review types.

Reliability therefore needs to be demonstrated using records representative of the review's actual eligibility challenges.

How Should Researchers Measure Eligibility Errors?

Evaluation should distinguish studies correctly retained, studies incorrectly excluded, ineligible studies correctly excluded, and ineligible studies unnecessarily retained.

False exclusions are especially consequential because they can remove potentially relevant evidence. Overall accuracy may obscure these errors when most retrieved records are ineligible.

Researchers should examine sensitivity or recall, specificity, precision, and the characteristics of misclassified studies. The appropriate metrics depend on the classification task and the review's decision rules.

The detailed procedures for measuring AI screening errors deserve separate attention because a favorable percentage can conceal important methodological weaknesses.

Can AI Replace Human Eligibility Decisions?

Generative AI may assist with preliminary classifications, but this does not automatically justify replacing independent human assessment.

The Cochrane Handbook requires at least two people working independently to determine final study eligibility for Cochrane intervention reviews, with a predefined process for resolving disagreements. Other review organizations may specify different requirements.

Researchers should not assume that a language model satisfies an independent human reviewer requirement simply because it produces a separate classification.

Any proposed substitution of AI for a second reviewer requires methodological justification, validation, and consideration of the applicable review standard.

Applying Eligibility Criteria to an Educational Technology Study

What Should the Final Decision Be?

Assuming the reported information is verified and the criteria are defined as described, the study would be excluded because it fails the intervention and study-design requirements.

The presence of an eligible population and outcome does not compensate for failure to meet another mandatory criterion.

For transparent reporting, the team should apply its predefined procedure for assigning exclusion reasons. If only one primary reason is recorded, the selected reason should follow that procedure rather than whichever criterion the AI happens to mention first.

The example also demonstrates why an AI model should not simply search for the words "automated feedback" or "random." The actual intervention mechanism and allocation procedure determine eligibility.

Common Misconceptions About AI Eligibility Decisions

Make Eligibility Criteria Explicit Before Evaluating AI

The most useful preparation is to translate broad inclusion and exclusion statements into operational decision rules. Specify what evidence would satisfy each criterion, what would clearly fail it, and what would require further investigation.

Then evaluate AI using representative records, including difficult borderline cases. A model that performs well on obvious exclusions may still make consequential mistakes when studies are similar but methodologically distinct.

Researchers should preserve criterion-specific decisions, source evidence, human corrections, and records of disagreement. An audit of AI decisions can help identify recurring errors, although the adequacy of sampling depends on the intended use and consequences of missed classifications.

Material AI involvement should also be documented in the review methods so readers can understand which decisions were assisted, how reliability was evaluated, and who made the final determinations.

Before Allowing AI to Apply Eligibility Criteria

Frequently Asked Questions About AI and Eligibility Criteria

Reliable Eligibility Assessment Requires More Than Following Instructions

How to Cite This Guide

This guide is intended to be read, shared, and used in research, teaching, and academic work. If you draw on its ideas, explanations, or other content, please acknowledge the source by citing the guide. Doing so gives appropriate credit and helps your readers locate the original resource.