Can Generative AI Replace a Second Human Reviewer in a Systematic Review?

Can One Researcher and One AI Model Replace Two Independent Reviewers?

Systematic reviews often use two independent reviewers to reduce mistakes and minimize the influence of individual judgments during study selection. However, recruiting and coordinating a second reviewer can be difficult, particularly for researchers working with limited resources.

Generative AI offers an appealing possibility. One researcher could screen the records while a language model independently produces another set of inclusion and exclusion decisions. Disagreements could then be examined, potentially reducing the need for a second human screener.

But independence is not simply a matter of producing two separate answers. The question is whether AI can provide sufficiently reliable decisions, whether its errors differ meaningfully from human errors, and whether replacing a person complies with the review's methodological standards.

Not as a General Replacement for Required Independent Human Review

What It Really Means to Replace a Second Systematic Reviewer

Why Do Systematic Reviews Use Two Independent Reviewers?

Independent review reduces the risk that study-selection decisions depend entirely on one person's interpretation of the eligibility criteria. Reviewers may overlook relevant information, misunderstand methodological details, or apply criteria inconsistently.

When two reviewers independently assess the same records, disagreements can reveal uncertainties that require further examination. A predefined resolution process, often involving discussion or an additional reviewer, helps establish the final decision.

Dual review does not guarantee correctness. Two people can make the same mistake. Nevertheless, it provides an important procedural safeguard against individual errors and subjective selection decisions.

Replacing one reviewer therefore involves more than demonstrating that AI can classify records. Researchers must consider whether the alternative preserves the relevant safeguards.

What Does Cochrane Actually Require?

The Cochrane Handbook for Systematic Reviews of Interventions and the Methodological Expectations of Cochrane Intervention Reviews (MECIR) distinguish preliminary screening from final eligibility assessment.

MECIR standard C39 requires at least two people working independently to determine whether each study meets the eligibility criteria, with a predefined procedure for resolving disagreements.

The guidance also explains that independent duplicate title and abstract screening is desirable but not mandatory. A single person may undertake the initial screening, whereas final eligibility decisions based on potentially eligible full-text reports require independent assessment by at least two people. Cochrane MECIR standards C39–C42.

This is not a universal rule imposed on every type of evidence synthesis. Other review organizations may have different requirements. Researchers should identify the standards applicable to their review before deciding whether AI can substitute for a human screening role.

What Does Research Say About AI Replacing Human Reviewers?

Khraisha and colleagues (2024) evaluated GPT-4 against human decisions for title and abstract screening, full-text assessment, and data extraction across different literature types and languages.

The study found that GPT-4 achieved performance comparable to humans in some tasks, particularly full-text screening under specific prompting conditions. However, apparent accuracy was influenced by chance agreement and dataset imbalance. After these factors were considered, performance was substantially less favorable across several tasks.

The authors concluded that substantial caution was warranted when considering language models for systematic review work, while acknowledging promising performance under particular conditions. Their findings do not establish that generative AI can generally replace a human reviewer. Khraisha et al. (2024).

The distinction is important because a model's performance on one review dataset may not transfer to another with different eligibility criteria, terminology, languages, or study designs.

Why Is Agreement With a Human Reviewer Not Enough?

Suppose a researcher and an AI model agree on 95% of screening decisions. That sounds reassuring, but the percentage does not reveal which records were incorrectly excluded.

If most retrieved records are irrelevant, agreement may be high simply because both reviewers classify nearly everything as ineligible.

More importantly, agreement does not establish correctness. A human reviewer and AI system may share the same misunderstanding of a criterion.

Researchers therefore need to examine disagreement patterns and independently verified decisions rather than treating agreement as proof of methodological equivalence.

Why Are False Exclusions Especially Important?

A false exclusion occurs when a record that should be retained is incorrectly classified as ineligible.

This error can be more consequential than unnecessarily retaining an irrelevant record. A false inclusion at preliminary screening creates additional assessment work, while a false exclusion may prevent a relevant study from entering the evidence base.

When evaluating AI as a possible second reviewer, researchers should examine whether it detects studies that the first reviewer overlooks and whether it introduces additional missed studies.

The detailed evaluation of AI screening errors should include sensitivity, false exclusions, and the consequences of disagreement-resolution procedures.

Can AI and Human Reviewers Make the Same Mistakes?

Yes. A language model may reproduce common assumptions or misinterpretations, particularly when eligibility criteria are ambiguous.

For example, both a human reviewer and AI might incorrectly treat random sampling as evidence of randomized allocation. Their agreement would not make the study eligible for a review restricted to randomized trials.

AI may also rely on superficial terminology that resembles human shortcut reasoning. The degree to which AI errors are independent of human errors must be investigated rather than assumed.

This matters because the purpose of independent review is not merely to produce two classifications, but to reduce the likelihood that consequential mistakes remain undetected.

What About Using AI Only When Reviewers Disagree?

AI may help identify relevant passages or clarify the wording of an eligibility criterion when human reviewers disagree. It could also organize the evidence supporting each proposed classification.

However, a model-generated recommendation should not automatically settle the disagreement. The final decision must follow the review's established resolution procedure and applicable methodological standards.

Where Cochrane requires two independent human eligibility assessments, consulting AI during disagreement resolution does not remove that requirement.

Could AI Replace a Second Reviewer During Title and Abstract Screening?

This possibility requires a more nuanced answer because preliminary screening and final eligibility assessment have different requirements.

Under Cochrane guidance, initial title and abstract screening may be performed by one person. Consequently, a workflow involving one human screener and AI assistance is not necessarily a departure from the minimum human staffing requirement at that stage.

That does not establish that AI provides the same protection as an independent second human reviewer. Its added value should be assessed through actual screening performance and the number of eligible records missed or recovered.

Researchers should avoid describing such a workflow as equivalent to dual independent human screening unless that claim is supported by appropriate evidence.

Why Is Full-Text Screening Different?

Final full-text screening determines which studies are included in the review. It often requires interpretation of complex methodological information, examination of related reports, and careful handling of missing details.

For Cochrane intervention reviews, this stage requires at least two independent human assessments. A model cannot simply be counted as a second person under that requirement.

AI may still assist with locating full-text eligibility evidence or preparing provisional classifications, provided the final decision process complies with the applicable standards.

Other review types may permit different procedures, but researchers should justify any departure from established safeguards rather than assume that technological capability makes the departure acceptable.

How Could Researchers Evaluate a Human-AI Screening Workflow?

A defensible evaluation should compare the proposed workflow against independently established reference decisions.

It should examine not only the accuracy of AI classifications but also how the combined human-AI process behaves. For example, researchers should determine whether AI identifies records missed by the human screener, whether disagreements are resolved correctly, and whether the workflow reduces labor without increasing consequential exclusions.

Evaluation should use representative records, including difficult borderline cases, and should distinguish preliminary screening from final eligibility assessment.

Researchers should also consider the time needed to verify AI outputs. A workflow that generates classifications quickly but requires extensive checking may offer little practical efficiency.

Comparing One Human Reviewer Plus AI With Independent Human Screening

Suppose the combined workflow retains a record whenever either the human reviewer or AI recommends retention. Under this rule, 78 of the 80 reference-positive records are retained, while two are missed by both.

Does This Prove AI Can Replace the Second Reviewer?

No. The combined workflow performs better than either individual classifier on this particular measure, but it still misses two potentially eligible records.

The result also does not show how many ineligible records were unnecessarily retained or how much time the combined process required. Nor does it establish that the performance would generalize to another review.

Most importantly, the example concerns preliminary screening. It does not justify replacing the two independent human assessments required for final full-text eligibility decisions under Cochrane standards.

The example illustrates that human-AI collaboration may improve some performance measures without becoming equivalent to two independent human reviewers. The distinction between better classification performance and methodological compliance should remain explicit.

Common Misconceptions About AI as a Second Reviewer

Use AI to Strengthen Screening Without Misrepresenting Reviewer Independence

Before considering a human-AI workflow, identify the stage of screening and the applicable methodological standard. The answer may differ between preliminary title and abstract screening and final full-text eligibility decisions.

Where AI assistance is permitted, evaluate its contribution using independently verified reference decisions and a predefined process for handling disagreements.

How Should a Human-AI Workflow Be Audited?

Researchers should examine both agreement and disagreement cases. Agreement cases matter because shared errors can remain invisible when only disagreements are reviewed.

Validation should include representative eligible studies, difficult exclusions, and records with incomplete information. The sample size and verification intensity should reflect the consequences of missed studies.

A structured audit of AI screening decisions can help identify systematic weaknesses, but a small convenience sample cannot establish that the workflow is free from consequential errors.

How Should the Workflow Be Reported?

Describe how many human reviewers participated, which tasks AI performed, how independent decisions were generated, how disagreements were resolved, and how AI performance was evaluated.

Do not use phrases such as "two independent reviewers screened all studies" if the second classifier was an AI system.

Transparent reporting of AI involvement should allow readers to understand exactly which decisions were made by humans and which were generated or assisted by AI.

Before Using AI as a Screening Partner

Frequently Asked Questions About AI Replacing a Second Reviewer

AI Can Assist a Reviewer, but It Is Not Automatically Another Reviewer

How to Cite This Guide

This guide is intended to be read, shared, and used in research, teaching, and academic work. If you draw on its ideas, explanations, or other content, please acknowledge the source by citing the guide. Doing so gives appropriate credit and helps your readers locate the original resource.