Can Generative AI Assess Certainty of Evidence in a Systematic Review?

Can AI Determine How Much Confidence Researchers Should Place in Review Findings?

A systematic review may find that an intervention improves an outcome, but the estimated effect does not tell researchers how confident they should be in that conclusion. Studies may have methodological limitations, inconsistent findings, indirect evidence, or estimates too imprecise to support a clear interpretation.

Frameworks such as GRADE help researchers assess these concerns systematically. Generative AI can organize study results, summarize methodological limitations, and produce explanations resembling certainty-of-evidence judgments.

Yet assigning a certainty rating requires interpreting the body of evidence for a particular outcome. Can AI make those judgments reliably, or is it more appropriate as an assistant to researchers applying an established framework?

AI Can Support GRADE Assessments, but Certainty Requires Human Judgment

How Generative AI Might Support GRADE Certainty Assessment

What Is Certainty of Evidence?

Certainty of evidence expresses how much confidence researchers can place in an estimate or conclusion derived from a body of research. In intervention reviews, the GRADE approach considers confidence that the true effect falls within a specified range or on a particular side of a decision threshold.

GRADE uses four certainty categories: high, moderate, low, and very low. These categories describe the certainty of the evidence for an outcome, not whether the intervention itself is good or bad.

A systematic review may therefore report moderate-certainty evidence for one outcome and low-certainty evidence for another, even when both outcomes are investigated by many of the same studies.

The official GRADE Book and the Cochrane Handbook provide the methodological framework for these assessments. Current GRADE guidance emphasizes the relationship between certainty, effect estimates, and thresholds relevant to decision-making.

How Is Certainty of Evidence Different From Risk of Bias?

Risk of bias evaluates whether systematic errors may have distorted particular study results. Certainty of evidence considers the broader body of evidence supporting an outcome.

Risk of bias is one component of GRADE, but it is not the entire assessment. Even a collection of studies with low risk of bias may provide low-certainty evidence if the estimates are highly imprecise or the studies address a population substantially different from the review's target population.

Consequently, AI-generated risk-of-bias judgments should not automatically become final GRADE judgments. Errors in the underlying appraisal can propagate into the certainty assessment.

What Are the Five Main GRADE Domains?

For intervention effects, GRADE considers five domains that may lower certainty: risk of bias, inconsistency, indirectness, imprecision, and publication bias or broader dissemination bias concerns.

These domains require different forms of reasoning. A model that can identify a confidence interval may still be unable to determine correctly whether it warrants downgrading for imprecision.

Can AI Apply GRADE Downgrading Rules Automatically?

Generative AI can be instructed to identify possible reasons for downgrading, but the decision depends on the official GRADE framework and the circumstances of the outcome being assessed.

For example, a wide confidence interval does not automatically justify a particular downgrade. Researchers must consider whether the interval crosses thresholds separating effects that would lead to different interpretations or decisions.

Similarly, statistical heterogeneity does not independently determine the inconsistency judgment. Reviewers need to consider the direction and magnitude of effects, overlap of confidence intervals, possible explanations for differences, and other relevant information.

A language model may produce a confident recommendation without correctly integrating these considerations.

Researchers should therefore treat AI-generated downgrading proposals as provisional until the underlying evidence and decision rules have been verified.

Does Every Randomized Trial Automatically Produce High-Certainty Evidence?

No. Under the conventional GRADE approach for intervention reviews, a body of evidence from randomized trials generally begins at high certainty. That initial rating can be reduced when concerns arise across the relevant domains.

Non-randomized intervention evidence traditionally begins at low certainty, although important qualifications apply. When non-randomized studies are evaluated using ROBINS-I, GRADE guidance permits an initial high-certainty starting point followed by assessment of relevant bias concerns and other domains.

Researchers should follow the applicable GRADE approach rather than using a rigid rule that all observational evidence must always begin at low certainty.

AI systems may reproduce simplified explanations of GRADE that overlook these distinctions. Official methodological guidance should take precedence over generated summaries.

Why Must Certainty Be Assessed Separately for Each Outcome?

A body of evidence may be convincing for one outcome but much less informative for another.

Consider a review of AI-supported formative feedback. Several randomized trials may provide relatively consistent estimates of academic achievement, while only a few studies report student anxiety, with substantial uncertainty around their estimates.

Assigning one overall certainty rating to the entire review would obscure this difference.

GRADE instead evaluates certainty at the outcome level for the relevant comparison. The judgment should reflect the evidence supporting that particular result.

Can AI Create a GRADE Summary of Findings Table?

Generative AI may assist with organizing verified effect estimates, participant counts, outcome definitions, and explanatory notes into a draft table.

However, a Summary of Findings table must accurately represent the evidence and the researchers' justified certainty judgments. A model should not invent missing estimates, infer downgrading decisions from incomplete information, or assign ratings solely to complete the table.

GRADEpro GDT provides dedicated functionality for preparing GRADE assessments and Summary of Findings tables. Researchers should consult the official software documentation and methodological guidance rather than assuming that an AI-generated table is equivalent to a properly completed GRADE assessment.

What Does Current Evidence Establish About AI-Generated GRADE Ratings?

Generative AI has been investigated across several evidence-synthesis activities, including screening, extraction, and methodological assessment. However, performance in these tasks does not establish reliability for GRADE certainty judgments.

GRADE requires integrated assessment across studies and outcomes, often involving uncertainty, context-specific thresholds, and interpretation of multiple domains. Evidence that a model can summarize individual articles or reproduce risk-of-bias terminology is insufficient to establish that it can assign valid certainty ratings.

Researchers should seek direct validation of the particular model and GRADE workflow. Without such evidence, autonomous certainty ratings should be regarded as unverified proposals rather than established assessments.

What Human Oversight Does GRADE Require?

The Cochrane Handbook's MECIR standard C74 requires certainty assessment for important outcomes using the relevant GRADE considerations. Standard C75 requires authors to justify and document their certainty judgments.

Cochrane guidance also indicates that, ideally, two people should independently assess certainty and reach consensus on downgrading decisions.

This distinction matters: documenting a model-generated rating does not satisfy the requirement to justify the rating. Researchers must be able to explain how the evidence supports each decision.

Other organizations may have additional or different expectations, so researchers should identify the applicable standards before incorporating AI into the assessment process.

Using AI to Prepare a GRADE Assessment of an Educational Intervention

Suppose the verified synthesis indicates a standardized mean difference of 0.30, with a 95% confidence interval from 0.05 to 0.55. This represents a positive estimated effect, although its practical importance depends on the context and thresholds selected by the review team.

How Could AI Help Without Making the Final Rating?

What Would a Defensible Interpretation Look Like?

Suppose the reviewers determine that risk-of-bias concerns justify downgrading the initially high-certainty randomized evidence by one level, while the other domains do not warrant further downgrading. The final rating would be moderate certainty.

This would mean that the reviewers have moderate confidence in the estimated effect, subject to the relevant GRADE interpretation and decision thresholds. It would not mean that the intervention is moderately effective or that 75% of the findings are correct.

The hypothetical rating depends on the reviewers' actual domain judgments. It should not be treated as a predetermined conclusion from the example's effect estimate alone.

Common Misconceptions About AI and GRADE Assessment

Use AI to Organize GRADE Evidence, Not to Manufacture Certainty

Begin with verified synthesis results and completed study-level appraisals. Identify the important outcomes and the decision thresholds relevant to interpreting their effects.

AI can then help organize the information required for each GRADE domain, but the research team should make and justify the final judgments using the official methodology.

How Should AI-Assisted GRADE Assessments Be Evaluated?

Where researchers wish to validate AI-generated certainty judgments, comparisons should use independently established reference assessments and examine agreement at the domain level as well as the final rating.

Particular attention should be paid to whether AI misses reasons for downgrading, applies inappropriate starting ratings, or misinterprets imprecision and indirectness.

Simple agreement percentages may conceal systematic errors. An audit of AI-generated decisions should include challenging outcomes and require reviewers to examine the reasoning behind disagreements.

Any material AI involvement should be documented, including the system used, tasks performed, source information supplied, human verification procedures, and limitations. This supports transparent reporting of AI-assisted evidence synthesis.

Before Accepting AI-Assisted GRADE Assessments

Frequently Asked Questions About AI and Certainty of Evidence

AI Can Help Organize Evidence, but Certainty Must Be Justified

How to Cite This Guide

This guide is intended to be read, shared, and used in research, teaching, and academic work. If you draw on its ideas, explanations, or other content, please acknowledge the source by citing the guide. Doing so gives appropriate credit and helps your readers locate the original resource.