Can Generative AI Extract Data From Studies Included in a Systematic Review?

Can AI Read Included Studies and Populate Your Data Extraction Table?

After selecting studies for a systematic review, researchers must collect information about their methods, participants, interventions, outcomes, and findings. These details are usually recorded in structured extraction forms and subsequently used to describe the evidence or calculate effect estimates.

Generative AI can read research reports and produce tables containing apparently relevant information within seconds. This makes it attractive for reviews involving dozens or hundreds of included studies.

The difficulty is that an extraction table must represent what the original studies actually reported. A model that confuses two intervention groups, selects the wrong follow-up measurement, or invents a missing numerical value can introduce errors that influence the synthesis.

Yes, but AI-Extracted Data Must Be Verified Against the Original Studies

How AI-Assisted Data Extraction Works and Where Errors Arise

What Is Data Extraction in a Systematic Review?

Data extraction is the systematic collection of predefined information from included studies. Researchers use extraction forms to record details needed to describe the studies, assess their methods, compare their findings, and conduct the planned synthesis.

The Cochrane Handbook emphasizes collecting information in a structured and unambiguous format that faithfully represents the original reports. Extraction forms should be developed and piloted before routine use.

Importantly, the unit of interest is generally the study rather than the publication. Several reports may describe the same investigation, and information from those reports may need to be combined or reconciled.

What Information Can Generative AI Extract?

AI can be instructed to identify particular fields in a research article and return them in a predefined structure. The requested fields should correspond to the review's actual data collection form rather than a generic list generated by the model.

These are potential extraction targets, not equally reliable AI applications. A model may accurately identify a publication year while incorrectly extracting the standard deviation needed for a meta-analysis.

What Does Published Research Show About Extraction Accuracy?

Research indicates that generative AI performance differs substantially by variable type, study characteristics, and evaluation method.

A systematic review of generative AI in evidence synthesis, based on searches conducted in January 2025, identified five comparative studies evaluating data extraction. Incorrect extractions ranged from 4% to 31%, with a median of 14% across the reported evaluations. These figures describe heterogeneous studies rather than a universal error rate. The published review also identified methodological limitations in the available evidence.

More directly relevant to education and behavioral research, Jansen and colleagues (2025) examined generative AI extraction using 22 systematic review databases from Psychological Bulletin. Eight language models extracted 312,329 data points from 2,179 studies across 186 variables.

The researchers found that accuracy was unacceptable across all evaluated metrics for 20% of variables. Only 12% of variables achieved excellent accuracy across all metrics. Performance varied more between variables than between models, and extraction was generally more accurate for contextual and moderator variables than for information required to calculate effect sizes. Jansen et al. (2025).

These findings challenge the assumption that choosing a more advanced language model automatically solves extraction reliability. The type of information being extracted may matter at least as much as the model selected.

Why Are Numerical Results Particularly Vulnerable?

Outcome data often involve several groups, measurement periods, statistical estimates, and denominators. A model must identify not only a number but also what that number represents.

For example, an article may report baseline and post-intervention achievement scores for both intervention and control groups. Extracting the correct mean but assigning it to the wrong group produces an inaccurate result even though the numerical value appears in the source.

Other potential errors include confusing standard deviations with standard errors, selecting an adjusted estimate when an unadjusted value was required, or mixing intention-to-treat and complete-case analyses.

These errors can affect effect-size calculations and therefore the conclusions of a meta-analysis.

How Should AI Handle Missing Information?

Generative AI may produce a plausible value even when the requested information is not explicitly reported. Researchers should therefore define how the system must handle unavailable or ambiguous data.

Useful distinctions include information explicitly reported, information derived through a documented calculation, information requiring interpretation, and information not reported in the available source.

These categories should not be merged. A missing standard deviation, for example, should not be replaced with an invented estimate. Where calculation or imputation is methodologically appropriate, the procedure must be justified and documented separately.

The Cochrane Handbook recommends retaining the provenance of extracted data and distinguishing original reported values from calculated or converted values.

Can AI Extract Information From Tables and Figures?

Some AI systems can interpret tables or image-based document content, but performance depends on the tool and the quality of the source representation.

PDF extraction may disrupt table columns, separate values from their headings, or omit footnotes that qualify the results. Figures may require specialized extraction procedures, particularly when numerical values are not provided directly.

Researchers should inspect the original table or figure rather than assuming that an AI-generated representation preserves its structure correctly.

When the model cannot reliably access the necessary information, manual extraction or a suitable specialized tool may be preferable.

What If Several Publications Report the Same Study?

A single investigation may produce a protocol, primary results article, secondary analysis, and follow-up report. These publications may contain overlapping or complementary information.

Data extraction should account for the underlying study rather than treating every report as an independent investigation. Otherwise, participants or results may be counted more than once.

The Cochrane Handbook describes approaches for extracting information from multiple reports and reconciling discrepancies between sources.

Generative AI may help compare reported details, but identifying overlapping study populations requires careful verification of study identifiers, recruitment periods, participants, and report relationships.

Can AI Replace Independent Human Data Extraction?

The Cochrane Handbook recommends independent extraction by more than one person to reduce errors. Its methodological expectations distinguish between study characteristics and outcome data.

For Cochrane intervention reviews, independent duplicate extraction of study characteristics is highly desirable, while independent extraction of outcome data by at least two people is mandatory under MECIR standard C46. A predefined disagreement-resolution process is also required.

This distinction is important because outcome data feed directly into evidence synthesis. A numerical extraction error may alter an effect estimate or change the interpretation of the findings.

Researchers should not assume that a language model automatically qualifies as one of the required independent human extractors. AI-generated entries may be used as provisional assistance, but the applicable extraction standard still governs the final workflow.

How Can Researchers Verify AI-Extracted Information?

A defensible workflow begins with a piloted extraction form and clear definitions for every field. The AI system is then instructed to populate those fields using only the supplied study reports.

For each entry, the model may be asked to identify the supporting passage, table, page, or source location. Researchers must verify that the cited material exists and supports the extracted value.

Verification should also distinguish between an incorrect value, an omitted value, a correct value assigned to the wrong field, and an unsupported inference.

Where AI performance is being evaluated, compare its entries against an independently established reference dataset. Record errors by variable type rather than relying solely on one overall accuracy percentage.

The adequacy of a sample-based audit depends on the intended use, the frequency of errors, and their consequences. A small sample of straightforward bibliographic fields cannot establish the reliability of numerical outcome extraction.

Extracting Achievement Results From an Educational Technology Study

What Went Wrong?

The AI correctly extracted the sample sizes and means but reported 6.5 instead of 8.0 for the intervention group's standard deviation. Suppose 6.5 was the standard deviation of a baseline measure elsewhere in the article.

The value is real, but it belongs to a different measurement occasion. This is an extraction error, not a harmless formatting difference.

If the incorrect standard deviation were used in a standardized mean difference calculation, the resulting effect estimate could differ from the estimate calculated using the correct data.

The example demonstrates why a mostly correct extraction table can still be unsuitable for direct analysis. The accuracy of each consequential field matters, especially when numerical values determine effect estimates.

Common Misconceptions About AI Data Extraction

Use AI for Provisional Extraction, Not Unverified Evidence

The most defensible approach is to treat AI-generated entries as candidate data requiring confirmation. Begin with a piloted extraction form, specify every variable clearly, and require source traceability.

Different extraction fields may justify different levels of scrutiny. Bibliographic and descriptive fields can still contain errors, but numerical outcomes, group assignments, and statistical measures are especially consequential because they may directly affect the synthesis.

What Should Be Documented?

Preserve the extraction form, variable definitions, source documents, AI instructions, model identification, provisional entries, human corrections, and final verified dataset.

Where information has been calculated or transformed, record the original values and the procedure used. Where discrepancies arise between reports, document how they were resolved.

These records support reproducibility and allow researchers to provide transparent reporting of AI-assisted extraction.

Before Using AI-Extracted Data in a Systematic Review

Frequently Asked Questions About AI Data Extraction

AI Can Prepare Extraction Data, but the Source Must Remain Authoritative

How to Cite This Guide

This guide is intended to be read, shared, and used in research, teaching, and academic work. If you draw on its ideas, explanations, or other content, please acknowledge the source by citing the guide. Doing so gives appropriate credit and helps your readers locate the original resource.