03 · What You Need to Know
How Generative AI Can Assist at Different Review Stages
First, Distinguish AI Assistance From Automated Decision-Making
Generative AI may help researchers complete an activity without independently making the final methodological decision. For example, a model could identify passages describing randomization while a reviewer determines whether those passages support a particular risk-of-bias judgment.
This distinction is especially important because systematic review tasks differ in their consequences. A formatting mistake may be corrected without changing the evidence base. An incorrect exclusion decision may prevent a relevant study from being considered at all.
Researchers should also distinguish generative AI from conventional automation. Bibliographic matching algorithms, machine-learning screening classifiers, and statistical software do not necessarily use generative language models. Evidence about one technology should not automatically be generalized to another.
Task 1: Developing Research Questions and Search Strategies
Generative AI can suggest alternative terminology, identify possible synonyms, and help organize preliminary concepts into search blocks. It may also explain database-specific search syntax or identify ambiguities in a draft question.
For example, a review concerning AI-supported feedback in higher education might consider terms such as automated feedback, intelligent tutoring, generative feedback, and computer-assisted formative assessment. AI could suggest candidate expressions that researchers subsequently evaluate.
However, a language model may overlook important terminology, introduce irrelevant concepts, or construct syntactically incorrect search strings. Search strategies must be checked against the requirements of each database and, where appropriate, peer reviewed using established approaches such as PRESS.
Conversational AI should not be assumed to provide comprehensive literature retrieval merely because it can name apparently relevant studies.
Task 2: Organizing Records and Identifying Duplicates
AI may assist in identifying records that appear to describe the same publication, particularly when titles, author names, or other metadata differ slightly between databases.
Nevertheless, deduplication is primarily a structured bibliographic matching problem. Established reference-management and systematic review tools may be more suitable for routine matching, while AI might help flag ambiguous cases for examination.
The important distinction is between duplicate records and multiple reports of the same underlying study. Removing one duplicate citation is not equivalent to deciding that two separate publications represent redundant evidence.
Researchers should therefore evaluate whether AI-assisted deduplication adds value beyond established procedures without incorrectly removing unique records.
Task 3: Screening Titles and Abstracts
Generative AI can classify titles and abstracts against predefined eligibility criteria. A model might label a record as potentially eligible, ineligible, or requiring further examination, while also identifying the text supporting its recommendation.
This application is attractive when searches retrieve thousands of records. Yet incorrect exclusions are particularly consequential because excluded studies may never receive full-text assessment.
Research examining large language models in screening has reported variable performance across datasets and tasks. High overall accuracy does not necessarily indicate that eligible studies are being retained reliably, especially when most retrieved records are irrelevant.
For AI-assisted title and abstract screening, researchers should assess sensitivity or recall, examine false exclusions, and establish appropriate human verification procedures before relying on automated decisions.
Task 4: Screening Full-Text Articles
Full-text screening requires more detailed judgments than title and abstract screening. Eligibility may depend on a study's design, participant characteristics, intervention components, comparison conditions, outcomes, or publication type.
Generative AI can locate candidate evidence within full-text reports and propose exclusion reasons. However, long documents, supplementary files, complex tables, and ambiguous reporting can make these judgments difficult.
Researchers should verify proposed exclusions against the original articles. In particular, absence of information from an AI response should not automatically be interpreted as evidence that the information is absent from the study.
Where AI supports full-text eligibility assessment, the process should preserve traceable reasons for exclusion and an appropriate mechanism for resolving uncertainty.
Task 5: Extracting Study Characteristics and Results
Generative AI can produce provisional extraction entries from study reports, including research design, sample characteristics, interventions, outcomes, and reported findings.
It may be particularly useful for organizing narrative descriptions into structured fields. Researchers can also ask models to identify source passages supporting each extracted item.
However, numerical extraction presents important risks. Models may confuse baseline and follow-up measurements, combine different study groups, misread tables, or report values that do not appear in the source.
A systematic review of generative AI applications in evidence synthesis reported incorrect data extraction rates ranging from 4% to 31% across the comparative studies examined. These heterogeneous findings do not establish a universal error rate, but they illustrate why extraction requires validation.
For AI-assisted data extraction, the source article should remain the authoritative record, not the model-generated spreadsheet.
Task 6: Supporting Risk-of-Bias Assessment
AI may help identify descriptions of randomization, allocation concealment, missing outcome data, measurement procedures, or selective reporting. Such passages can assist reviewers applying an appropriate risk-of-bias instrument.
Locating relevant text, however, is not equivalent to interpreting methodological bias. A study might report that participants were randomized without providing enough information to establish whether allocation concealment was adequate.
Risk-of-bias judgments should therefore be made by qualified reviewers using the relevant assessment framework. AI-generated ratings require task-specific validation and should not be treated as authoritative merely because the model provides a detailed explanation.
Task 7: Supporting Certainty-of-Evidence Assessment
Certainty assessment considers confidence in the evidence for particular outcomes. Under GRADE, judgments may involve risk of bias, inconsistency, indirectness, imprecision, and publication bias.
AI might help organize the information reviewers need, identify reported limitations, or draft preliminary explanations. Nevertheless, the final assessment depends on interpreting the body of evidence rather than merely extracting statements from individual papers.
Researchers should be especially cautious about using AI to make autonomous certainty-of-evidence judgments, because apparently reasonable explanations may conceal incorrect assumptions or incomplete consideration of the evidence.
Task 8: Supporting Evidence Synthesis
Generative AI can help organize verified findings by theme, intervention type, outcome, or study characteristic. It may also assist in drafting narrative descriptions from a researcher-approved synthesis table.
Yet synthesis involves more than summarizing each included study. Researchers must consider comparability, heterogeneity, methodological limitations, and the appropriateness of combining findings.
Where meta-analysis is appropriate, numerical calculations should be conducted using suitable statistical methods and software. A language model's ability to produce an effect estimate or forest-plot description does not establish that the calculation is correct.
AI-generated narrative synthesis should be checked for unsupported generalizations, omitted contradictory findings, and claims that exceed the available evidence.
Task 9: Preparing PRISMA Documentation and Reports
AI may assist with formatting verified study-selection counts, drafting descriptions of documented procedures, and organizing information required by reporting checklists.
For example, it could help prepare a narrative explaining how records moved from identification through screening to inclusion, provided the numbers come from authentic review records.
It can also help organize information for a PRISMA flow diagram. However, it should not invent missing counts, infer undocumented exclusion reasons, or reconstruct a selection history that the researchers did not preserve.
PRISMA 2020 provides reporting guidance, not a blanket endorsement of any particular AI method. The review must accurately describe the procedures actually followed.
How Should Researchers Compare These Applications?
| Task |
Potential AI role |
Verification priority |
| Search preparation |
Suggest candidate terminology |
Search completeness and database syntax |
| Deduplication |
Flag possible duplicate records |
Incorrect removal of unique records |
| Title and abstract screening |
Recommend eligibility classifications |
False exclusions and ambiguous records |
| Full-text screening |
Locate eligibility evidence |
Accuracy of final decisions and reasons |
| Data extraction |
Generate provisional extraction entries |
Source fidelity and numerical accuracy |
| Risk-of-bias assessment |
Identify relevant methodological passages |
Interpretive judgments against the chosen tool |
| Certainty assessment |
Organize supporting evidence |
Outcome-level judgments and rationale |
| Evidence synthesis |
Organize and describe verified findings |
Unsupported conclusions and calculation errors |
| Reporting |
Format documented methods and results |
Consistency with the actual review record |
These are potential applications, not universally validated levels of performance. The appropriate degree of automation depends on the particular system, review, and validation evidence.
