How Should Researchers Measure AI Errors During Systematic Review Screening?

How Can You Tell Whether AI Is Missing Studies During Screening?

Suppose generative AI screens 2,000 records for a systematic review and correctly classifies 95% of them. That sounds impressive. But what if most of the records are irrelevant and the model incorrectly excludes several studies that should have proceeded to full-text assessment?

Overall accuracy alone cannot answer whether the screening procedure is sufficiently reliable. Researchers need to distinguish errors that create additional work from errors that remove potentially relevant evidence.

The methodological challenge is to measure these errors against trustworthy reference decisions and determine what the results mean for the review's completeness and credibility.

Measure False Exclusions, Not Just Overall Accuracy

Which Measures Reveal AI Screening Errors?

Start With a Trustworthy Reference Standard

Before calculating performance, researchers need a reference classification against which AI decisions can be compared. This is sometimes called a reference standard or ground truth, although the latter term should not imply that human judgments are infallible.

A defensible reference set may be established through independent human screening followed by a predefined disagreement-resolution procedure. The reviewers should apply the same eligibility criteria that the AI system is expected to use.

Reference decisions should ideally be established without access to the AI classifications being evaluated. Otherwise, the model's recommendations may influence the supposedly independent comparison.

Researchers should also specify what counts as a positive classification. During title and abstract screening, a positive usually means retaining a record for further assessment, not necessarily including the study in the final review.

What Are True Positives, False Positives, True Negatives, and False Negatives?

A confusion matrix compares AI classifications with reference decisions. For a binary screening evaluation, records are classified as either retained or excluded.

A true positive is a record AI correctly retains. A false positive is an ineligible record AI unnecessarily retains. A true negative is a record AI correctly excludes. A false negative is a record AI incorrectly excludes despite the reference decision indicating that it should be retained.

In systematic review screening, false negatives deserve particular attention because they can prevent potentially eligible studies from receiving further assessment.

Why Is Sensitivity or Recall So Important?

Sensitivity measures the proportion of reference-positive records correctly retained by AI. In information retrieval, the same measure is commonly called recall.

The result means that AI retained 90% of the records that should have advanced. It also missed 10%. It does not mean that 90% of all AI decisions were correct or that the resulting review would necessarily retain 90% of ultimately eligible studies.

The consequences depend on which records were missed. A false exclusion involving a highly relevant study may matter considerably even when the overall false-negative count is small.

What Is the False-Negative Rate?

The false-negative rate expresses the proportion of reference-positive records incorrectly excluded. It complements sensitivity.

The false-negative rate should not be confused with the proportion of AI exclusions that are wrong. That latter quantity uses a different denominator, FP-free exclusion counts: FN divided by FN plus TN.

What Does Specificity Measure?

Specificity measures how effectively AI identifies records that the reference standard classifies as ineligible.

A specificity of 90% means that 90% of the reference-negative records were correctly excluded. It does not establish how many eligible studies AI missed.

Higher specificity may reduce unnecessary full-text assessment, but maximizing specificity at the expense of sensitivity can be undesirable in a review where missing eligible studies is particularly consequential.

What Is Precision?

Precision measures the proportion of AI-retained records that the reference standard also classifies as records requiring retention. It is also called positive predictive value.

Half of the AI-retained records in this example are reference-positive. The other half require further assessment despite being reference-negative.

Precision is useful for understanding screening workload, but it depends partly on the proportion of eligible records in the dataset. A model may show different precision across reviews even when its sensitivity and specificity remain similar.

Why Can Overall Accuracy Be Misleading?

Accuracy measures the proportion of all records correctly classified.

This illustrates class imbalance. When most records are irrelevant, correctly excluding them can dominate the accuracy calculation and conceal missed eligible studies.

Researchers should therefore avoid describing a model as reliable solely because its overall accuracy is high.

What About F1 Score, Agreement, and Cohen's Kappa?

The F1 score combines precision and recall through their harmonic mean. It can help summarize the trade-off between unnecessary retention and missed eligible records, but it does not directly account for true negatives or the unequal consequences of errors.

Cohen's kappa adjusts observed agreement between two classifiers for agreement expected by chance under its assumptions. It may be informative when comparing AI and human decisions, although class imbalance can complicate interpretation.

Neither F1 nor kappa should replace examination of the confusion matrix and the actual records incorrectly excluded.

Agreement between AI and one human reviewer is also not equivalent to agreement with a reliable reference standard. Both may make the same mistake.

How Should Uncertain AI Classifications Be Evaluated?

Some screening workflows permit three categories: retain, exclude, and uncertain. These cannot be evaluated meaningfully using a binary confusion matrix until researchers specify how uncertainty affects the workflow.

For example, uncertain records may automatically advance to human assessment. Under that rule, they should not be treated as autonomous AI exclusions.

Researchers should report the number of uncertain classifications and the workload they create. Collapsing uncertainty into an arbitrary include-or-exclude category may conceal how the system actually operates.

Should Performance Be Measured for the AI Model or the Combined Workflow?

Both may be useful, but they answer different questions.

Model-level evaluation asks how accurately AI classifies records independently. Workflow-level evaluation asks how the complete process performs after applying human verification, disagreement resolution, or combined retention rules.

For example, if AI proposes exclusions but a human verifies every exclusion before removal, the final workflow may retain records that AI would otherwise have missed. Reporting only the final workflow's sensitivity would conceal the model's original errors.

Researchers should distinguish provisional AI classifications from final screening decisions when reporting AI-assisted methods.

What Does Published Research Suggest About Screening Performance?

Khraisha and colleagues (2024) evaluated GPT-4 on systematic review screening and extraction tasks across different literature types and languages. Their findings demonstrated that apparently favorable accuracy can coexist with less reassuring agreement measures and task-dependent performance.

A systematic review of generative AI in evidence synthesis published in Research Synthesis Methods also found substantial variation in screening errors across comparative evaluations.

These findings support using multiple measures and examining task-specific errors rather than assuming that one performance estimate generalizes across systematic reviews.

Calculating AI Screening Errors From a Confusion Matrix

How Should These Results Be Interpreted?

AI correctly retains 90 of the 100 reference-positive records, but incorrectly excludes 10. It also retains 90 records that the reference process would exclude, increasing the number requiring further assessment.

The overall accuracy of 90% does not resolve the principal methodological concern: ten potentially eligible records would be lost if AI exclusions were accepted without verification.

The precision of 50% means that half of AI-retained records are reference-positive. That is relevant to workload, but it does not compensate for the false exclusions.

The example does not establish a universal minimum sensitivity for AI screening. An acceptable performance threshold must be justified in relation to the review's purpose, methodological requirements, and the consequences of missed evidence.

Common Misinterpretations of AI Screening Performance

Evaluate the Screening Workflow Against the Errors That Matter Most

Before testing AI, define the positive class and the reference decision procedure. Decide whether the evaluation concerns title and abstract screening, full-text eligibility assessment, or a combined human-AI workflow.

Use a dataset that represents the actual review, including ambiguous records and difficult eligibility distinctions. Where the evaluation uses a sample, report its size and composition so readers can judge the precision and applicability of the results.

How Much Confidence Can You Place in the Estimated Metrics?

Performance percentages are estimates based on the evaluated records. Their precision depends on sample size, the number of reference-positive records, and the sampling design.

For example, observing no false exclusions among ten eligible records provides much weaker evidence of high sensitivity than observing none among several hundred eligible records. Confidence intervals can help communicate this uncertainty, provided the sampling assumptions are appropriate.

Researchers should also distinguish a random or representative validation sample from a deliberately enriched test set containing additional eligible records. Enrichment may help investigate difficult cases but can change the interpretation of precision and workload estimates.

When conducting a manual audit of AI decisions, sampling choices should therefore be documented rather than treated as an inconsequential administrative detail.

What Should Be Reported?

Report the model and relevant configuration, screening task, eligibility criteria, reference-standard procedure, dataset size, class distribution, confusion matrix, performance measures, and human verification process.

Explain how uncertain classifications were handled and whether reported results describe the AI model alone or the final human-AI workflow.

These details make performance claims interpretable and allow readers to assess whether the reported reliability supports the role assigned to AI.

Before Reporting AI Screening Accuracy

Frequently Asked Questions About AI Screening Error Metrics

Screening Reliability Depends on Which Errors AI Makes

How to Cite This Guide

This guide is intended to be read, shared, and used in research, teaching, and academic work. If you draw on its ideas, explanations, or other content, please acknowledge the source by citing the guide. Doing so gives appropriate credit and helps your readers locate the original resource.