Can Generative AI Screen Titles and Abstracts for a Systematic Review?

Can AI Decide Which Titles and Abstracts Deserve Further Review?

A systematic review search may retrieve hundreds or thousands of records, many of which are clearly irrelevant. Researchers must examine their titles and abstracts to identify potentially eligible studies before obtaining and assessing full-text reports.

This can be one of the most labor-intensive stages of evidence synthesis. Generative AI appears well suited to the task because it can read text, follow instructions, and classify records according to specified criteria.

But title and abstract screening is not simply a sorting exercise. Incorrectly excluding an eligible study may prevent it from entering the review altogether. The central question is whether AI can assist with screening while keeping that risk acceptably low.

Yes, but Screening Recommendations Must Be Validated

How Generative AI Screening Works and What Determines Its Reliability

What Happens During Title and Abstract Screening?

Title and abstract screening is an initial eligibility assessment conducted after search results have been collected and duplicate records addressed. Reviewers examine available bibliographic information to determine which reports may satisfy the review's predefined inclusion criteria.

At this stage, the objective is generally to identify records requiring further examination rather than to establish final study eligibility. Abstracts often omit details needed for a definitive decision.

For example, a review examining randomized trials of AI-supported formative feedback in universities might retrieve an abstract describing an educational technology intervention without clearly identifying whether participants were university students or whether allocation was randomized.

Such uncertainty may justify retaining the record for full-text assessment rather than excluding it prematurely.

How Does Generative AI Classify Titles and Abstracts?

A generative AI screening workflow typically supplies the model with the review's eligibility criteria and the title and abstract of each record. The model is instructed to compare the available information with those criteria and return a structured recommendation.

Possible classifications include:

• Include for full-text assessment: The available information suggests that the record may satisfy the criteria.
• Exclude: The available information clearly indicates that the record does not satisfy one or more criteria.
• Uncertain: The title and abstract do not provide enough information to make a defensible determination.

These categories are practical workflow choices, not universal classifications required by every review organization. Researchers should define their decision rules before screening begins.

Some workflows also request an explanation or supporting text. Such explanations can help reviewers inspect decisions, but they do not independently establish that a classification is correct.

Why Do Explicit Eligibility Criteria Matter?

AI screening depends heavily on how eligibility requirements are expressed. Broad instructions such as "include relevant studies about artificial intelligence in education" leave substantial room for interpretation.

A more operational definition might specify university students, a particular intervention, eligible study designs, and required outcomes. These criteria should reflect the registered or otherwise documented review protocol.

Even precise instructions cannot guarantee reliable decisions. Abstracts may use unfamiliar terminology, describe several populations, or omit essential details.

Researchers therefore need to determine whether AI can apply their inclusion and exclusion criteria consistently, rather than assuming that a well-written prompt eliminates screening errors.

What Does Published Research Show About AI Screening Performance?

Comparative studies of large language models have reported mixed results. Khraisha and colleagues (2024) evaluated GPT-4 for screening and extraction across different literature types and languages, illustrating how performance can vary according to the characteristics of the evidence and the evaluation approach.

A later systematic review of generative AI applications in evidence synthesis, published in Research Synthesis Methods, identified substantial variation in incorrect exclusion decisions across the studies examined. Reported rates ranged from 1% to 83%, although the underlying tasks, datasets, and methods differed considerably.

These findings should not be interpreted as a universal error range for every model or systematic review. They do indicate that screening reliability cannot safely be inferred from the model's general language capabilities.

Performance should be evaluated on material resembling the records, eligibility criteria, and disciplinary terminology of the actual review.

Why Is Overall Accuracy Not Enough?

Most records retrieved during a systematic review may ultimately be excluded. This creates a class imbalance problem: a screening system can appear highly accurate simply by classifying most records as irrelevant.

Suppose a dataset contains 1,000 records, of which only 50 should proceed to full-text assessment. An AI system that excludes all 1,000 records would classify 950 correctly, producing an overall accuracy of 95%.

Yet it would miss every potentially eligible record. Its screening sensitivity would be zero.

A sensitivity of 90% means that the system retained 90% of records classified as potentially eligible in the reference set. It also means that 10% were missed. Whether that performance is acceptable depends on the review's requirements and the consequences of those omissions; it should not automatically be considered sufficient.

Researchers should examine screening error measures such as sensitivity, specificity, precision, and the absolute number and characteristics of missed records.

Can AI Replace a Second Human Screener?

Some researchers may consider replacing one of two independent reviewers with AI to reduce screening workload. However, the ability to produce classifications does not establish that AI provides equivalent protection against errors or bias.

Human reviewers can disagree, and agreement between AI and a reviewer does not necessarily mean their shared decision is correct. Both may misunderstand an eligibility criterion or overlook information.

The Cochrane Handbook discusses study-selection procedures and the importance of independent assessment. Any proposed replacement of a human reviewer with AI should therefore be evaluated against the methodological expectations of the review and supported by appropriate evidence.

Where established guidance requires independent human decisions, researchers should not assume that a generative AI classification automatically satisfies that requirement.

How Should Uncertain Records Be Handled?

When titles and abstracts lack sufficient information, retaining the record for further examination is generally safer than excluding it based on assumptions.

An AI system should not interpret the absence of a reported study characteristic as proof that the characteristic is absent. For instance, an abstract that does not mention randomization may still describe a randomized study.

Researchers can design screening instructions that require the model to identify uncertainty explicitly and avoid unsupported exclusion decisions. The effectiveness of those instructions must still be tested.

Records retained during preliminary screening proceed to full-text eligibility assessment, where additional information can be examined.

What Records Should Be Preserved?

AI-assisted screening should maintain a traceable record of each classification, the bibliographic record assessed, the criteria applied, and any human corrections or overrides.

Where feasible, researchers should also preserve the model identification, version information, instruction templates, screening dates, and relevant configuration details. Hosted AI systems may change, making exact reproduction difficult even when prompts are preserved.

These records support methodological transparency and allow investigators to examine whether AI errors affected the review's evidence base.

Testing AI Screening in a Review of AI-Supported Feedback

The results indicate that the system missed 10% of records that should have advanced to full-text assessment. Although its overall accuracy was relatively high, autonomous exclusion would have removed eight potentially eligible records.

The team could investigate why those records were missed, revise the workflow, and conduct further evaluation. The initial results alone would not justify allowing AI to make final exclusion decisions.

Notice that the reference set identifies records requiring full-text assessment, not necessarily studies that will ultimately be included in the systematic review. Final eligibility remains a separate determination.

Misconceptions About AI Title and Abstract Screening

Validate AI Screening Before Allowing It to Influence Exclusions

Begin with operational eligibility criteria and an independently established set of reference decisions. Evaluate AI performance using records representative of the review, including ambiguous abstracts and studies that are difficult to distinguish from ineligible material.

Pay particular attention to false exclusions. An apparently efficient workflow may be methodologically unacceptable if it misses relevant studies.

Researchers should also decide how AI-assisted classifications will be checked, how disagreements will be resolved, and how changes to the workflow will be documented.

When validation relies on sampling rather than checking all decisions, the audit procedure must be designed carefully enough to detect consequential errors. A small sample containing mostly obvious exclusions may provide little reassurance about missed eligible studies.

Before Using AI to Screen Titles and Abstracts

Frequently Asked Questions About AI Abstract Screening

AI Screening Is Useful Only When Eligible Studies Are Protected

How to Cite This Guide

This guide is intended to be read, shared, and used in research, teaching, and academic work. If you draw on its ideas, explanations, or other content, please acknowledge the source by citing the guide. Doing so gives appropriate credit and helps your readers locate the original resource.