Can Generative AI Help Deduplicate Systematic Review Search Results?

Can AI Recognize Duplicate References Across Different Databases?

When researchers search Scopus, Web of Science, PubMed, ERIC, and other databases, the same publication may appear several times. One database might provide a complete title and DOI, while another exports abbreviated author names, a slightly different title, or incomplete publication details.

These overlapping records must be managed before screening. Otherwise, reviewers may examine the same publication repeatedly, inflate record counts, or create confusion when documenting study selection.

Generative AI can compare text and recognize similarities, which raises a practical question: could it identify duplicates that conventional reference-management software misses, and would doing so be reliable enough for a systematic review?

AI Can Flag Possible Duplicates, but Removal Requires Verification

How Deduplication Works and Where Generative AI Might Help

What Is Deduplication in a Systematic Review?

Deduplication is the process of identifying and removing redundant bibliographic records retrieved from different searches or information sources. The purpose is to produce a set of unique records for subsequent screening while preserving the evidence needed to reconstruct the search process.

Suppose a journal article appears in three databases. Although the search retrieves three records, those records represent one publication. The researchers ordinarily retain one usable bibliographic record for screening and account for the removed duplicates in their search documentation.

Deduplication is different from determining whether multiple publications describe the same underlying study. The Cochrane Handbook distinguishes removing duplicate records of the same report from linking multiple reports of one study. Both processes matter, but they address different problems.

This distinction is particularly important when a study has a conference abstract, a full journal article, and a later follow-up report. Their titles may be similar, but each document may contain information needed for the review.

How Do Conventional Deduplication Methods Identify Matches?

Reference-management software typically compares bibliographic fields such as title, author, publication year, journal, volume, pages, DOI, or other identifiers. Some workflows apply several matching rules sequentially to accommodate missing or inconsistent metadata.

Bramer and colleagues (2016) described a structured EndNote deduplication method designed for systematic review search results. Their work illustrates how bibliographic matching can be improved through carefully configured comparison procedures rather than relying on a single exact-match rule.

Established approaches remain useful because their comparison logic can be specified and evaluated. A record with the same DOI as another record is a strong candidate for duplication, although identifiers and metadata still require checking when inconsistencies arise.

Exact matching is not always sufficient. Database exports may differ in punctuation, capitalization, abbreviated titles, special characters, author initials, or available identifiers. This is where more flexible matching methods may help.

What Could Generative AI Add to the Process?

Generative AI can compare records using the meaning and structure of their bibliographic information rather than relying exclusively on identical strings. For example, it might recognize that a title with an abbreviated phrase and another with its expanded wording could refer to the same publication.

A researcher could provide two candidate records and ask the model to identify matching and conflicting fields. The model could then produce a provisional classification such as likely duplicate, uncertain match, or apparently distinct record, accompanied by an explanation.

Such assistance may be useful when conventional software flags ambiguous pairs. However, the ability to produce a plausible comparison does not establish that generative AI is more accurate than established matching algorithms. Direct validation is necessary before claiming an improvement.

Which Bibliographic Differences Can Make Matching Difficult?

These examples illustrate potential matching problems, not evidence that a particular generative AI model can resolve all of them reliably.

What Are False Positives and False Negatives in Deduplication?

Deduplication involves two important types of error.

A false-positive duplicate match occurs when two distinct records are incorrectly classified as duplicates. If the researcher removes one, a unique publication may disappear from the screening dataset.

A false-negative duplicate match occurs when records referring to the same publication are not recognized as duplicates. Both remain in the dataset, creating additional screening work and potentially complicating record accounting.

False-positive matches are especially concerning because they can result in the loss of potentially relevant evidence. False negatives are also undesirable, but they may be discovered during subsequent screening or record management.

The consequences of these errors are not necessarily symmetrical. Researchers should therefore assess both types rather than relying on a single overall accuracy percentage.

Can AI Safely Remove Duplicates Automatically?

Automatic removal should not be adopted solely because a language model labels two records as duplicates. A defensible workflow needs reliable matching criteria, evidence of performance, and a mechanism for reviewing questionable decisions.

For records with consistent, verified identifiers and bibliographic details, established matching procedures may already be sufficient. AI assistance may be more relevant to uncertain pairs, where a researcher can examine the model's comparison alongside the original records.

Even then, the model should not be treated as an authoritative bibliographic database. It may overlook conflicting metadata or infer similarities that are not supported by the records.

A practical safeguard is to preserve the original search exports and maintain a recoverable record of every proposed removal. That allows researchers to reverse decisions when a false match is discovered.

How Does Deduplication Affect the PRISMA Flow Diagram?

PRISMA 2020 requires transparent reporting of the identification and selection process. Its flow diagram includes records removed before screening, including duplicates, where applicable.

Researchers therefore need reliable counts of the records identified and removed. If AI is involved, its decisions should be incorporated into the documented deduplication workflow rather than treated as an undocumented preliminary cleanup.

Importantly, deduplication counts should reflect actual record-management operations. Researchers should not ask a model to estimate how many duplicates were removed or reconstruct counts from incomplete information.

Accurate documentation is also necessary when preparing the PRISMA flow diagram.

Using AI to Examine Ambiguous Duplicate Records

The result means that 1,155 unique bibliographic records remain after the documented duplicate removals. It does not mean that 1,155 distinct studies have been identified, since different reports may still describe the same underlying study.

Nor does the example establish that AI improved deduplication accuracy. Demonstrating that would require comparing its decisions with a suitable reference standard and examining whether it contributed correct matches that established methods had missed.

Common Mistakes When Using AI for Deduplication

Use AI to Investigate Uncertain Matches, Not to Delete Records Blindly

For most systematic reviews, a sensible starting point is to use established bibliographic deduplication procedures and reserve additional examination for ambiguous cases. Generative AI may be considered as one possible aid to that examination.

The central requirement is to preserve the distinction between identifying a candidate match and authorizing removal. A candidate match is a proposition to be verified, not a final decision.

How Should AI-Assisted Deduplication Be Evaluated?

Where AI influences removal decisions, researchers should compare its classifications against independently verified duplicate and nonduplicate pairs. The evaluation should include difficult cases, not only obvious matches.

Useful outcomes include the number of correctly identified duplicates, false duplicate matches, missed duplicates, and cases requiring manual resolution. Researchers should also consider the additional time spent verifying AI suggestions.

If AI introduces more checking work than it saves, or produces unacceptable false matches, its use may not be justified. Efficiency should be demonstrated rather than presumed.

What Should Be Recorded?

Preserve original database exports, the number of records retrieved from each source, the deduplication procedure, removal counts, and the final screening dataset. Where AI materially contributes, record the system used, relevant instructions, the decisions it proposed, and how those decisions were verified.

These records support transparent reporting of AI-assisted review methods and make it possible to investigate errors discovered later.

Before Removing AI-Flagged Duplicate Records

Frequently Asked Questions About AI-Assisted Deduplication

AI May Help Find Duplicates, but Accuracy Matters More Than Removal Speed

How to Cite This Guide

This guide is intended to be read, shared, and used in research, teaching, and academic work. If you draw on its ideas, explanations, or other content, please acknowledge the source by citing the guide. Doing so gives appropriate credit and helps your readers locate the original resource.