Can Generative AI Perform Risk-of-Bias Assessment in a Systematic Review?

Can AI Judge Whether an Included Study Is at Risk of Bias?

After identifying eligible studies, systematic reviewers must evaluate whether limitations in their design, conduct, analysis, or reporting could distort the results. This assessment can be demanding because methodological details are often scattered across articles, supplementary materials, and study registrations.

Generative AI can locate passages about randomization, missing data, outcome measurement, and statistical analysis. It can also generate explanations that resemble formal critical appraisal judgments.

But identifying methodological information is not the same as determining whether a study's results are likely to be biased. Can researchers use AI to make these assessments reliably, or should its role remain limited to supporting human reviewers?

AI Can Support Risk-of-Bias Assessment, but Its Judgments Require Validation

What AI Can and Cannot Do During Risk-of-Bias Assessment

What Is Risk of Bias in a Systematic Review?

Risk of bias concerns the possibility that systematic errors in a study's design, conduct, analysis, or reporting have distorted its estimated results. It differs from random error, which concerns uncertainty arising from sampling variability and other chance processes.

Risk-of-bias assessment is not simply a judgment about whether a study is generally well written or methodologically sophisticated. It evaluates specific mechanisms through which bias could affect an outcome or estimated effect.

A randomized trial, for example, may use an appropriate allocation procedure but still have serious concerns arising from missing outcome data. Conversely, incomplete reporting of a methodological procedure does not necessarily establish that the procedure was performed incorrectly.

Researchers must distinguish what is known about the study from what remains uncertain.

Which Risk-of-Bias Tools Might AI Help Apply?

Different study designs require different assessment approaches. Two important frameworks in intervention reviews are Cochrane's RoB 2 tool for randomized trials and ROBINS-I for non-randomized studies of interventions.

These tools should not be treated as interchangeable. Their domains and signaling questions reflect different sources of potential bias.

Tool selection must follow the review question and study design. Researchers should consult the official instructions for the relevant instrument rather than asking AI to invent a generic risk-of-bias score.

Why Is RoB 2 More Than a Methodological Checklist?

RoB 2 evaluates risk of bias in a particular result from a randomized trial. It is not simply a single quality rating assigned to the publication as a whole.

Its five domains concern bias arising from the randomization process, deviations from intended interventions, missing outcome data, measurement of the outcome, and selection of the reported result.

The tool uses signaling questions and an algorithm to generate proposed judgments, which assessors consider alongside the relevant evidence. Domain-level judgments contribute to an overall risk-of-bias judgment for the result being assessed.

For example, the same trial could have different risk-of-bias judgments for a relatively objective outcome and a subjective self-reported outcome. An AI system that assigns one generic rating to the entire article may overlook this distinction.

Why Does ROBINS-I Require Different Reasoning?

ROBINS-I evaluates non-randomized intervention studies by considering how their results compare with those that would be expected from an appropriately specified hypothetical randomized trial.

This requires identifying important confounding factors, understanding how participants entered the study, and evaluating intervention classification and other potential sources of bias.

For example, a study comparing students who voluntarily used an adaptive learning platform with students who did not may be affected by pre-existing differences in motivation or academic preparation.

A model might recognize that the study reports a statistical adjustment for motivation. However, deciding whether that adjustment adequately addresses confounding requires substantive and methodological judgment.

Researchers should also check which version of ROBINS-I they are using, since the framework has undergone revisions. Instructions from different versions should not be mixed without justification.

What Can AI Reliably Contribute Without Making the Final Judgment?

One potentially useful application is evidence location. Researchers can ask AI to identify where a study describes allocation procedures, participant attrition, outcome measurement, or analysis decisions.

AI may also organize these passages under the domains of the selected appraisal tool, identify apparently missing information, and prepare provisional responses to signaling questions.

However, each source passage must be verified. A model may misquote a document, overlook a relevant appendix, or infer that a procedure occurred when the report merely states that it was planned.

It is useful to separate three operations: identifying evidence, interpreting the evidence against the tool's criteria, and assigning the final judgment. Successful performance on the first does not establish competence in the other two.

What Does Research Suggest About Automated Risk-of-Bias Assessment?

Research on automated risk-of-bias assessment predates contemporary generative AI. Marshall and colleagues developed RobotReviewer, a machine-learning system intended to assist with aspects of risk-of-bias assessment in randomized trials.

Such systems demonstrate that automated methods can identify relevant methodological language and generate provisional assessments. They do not establish that a conversational large language model can independently perform every domain-level judgment accurately.

Evaluations of generative AI in evidence synthesis have also identified variability across methodological tasks. The available evidence supports investigating task-specific applications, but it does not justify assuming that an AI-generated appraisal is equivalent to an independently verified human assessment.

Researchers should distinguish evidence supporting automated evidence identification from evidence supporting final appraisal decisions.

Why Can AI Misinterpret Methodological Reporting?

A study may state that participants were randomly assigned without explaining how the allocation sequence was generated or concealed. A model might treat the word "randomized" as sufficient evidence of a low risk of bias.

That conclusion may be premature. The relevant assessment tool may require additional information about the randomization process, baseline imbalances, or other circumstances.

Likewise, an AI model might infer that missing data were handled adequately because the authors used a recognized statistical method. Whether the method appropriately addresses potential bias depends on the nature and amount of missing data, the assumptions involved, and the result being assessed.

The central problem is that methodological terminology can sound reassuring without resolving the underlying bias question.

Can AI Produce a Single Risk-of-Bias Score?

Researchers should not ask AI to create an arbitrary numerical score unless the selected assessment instrument explicitly supports that form of scoring.

RoB 2 uses domain-level and overall judgments rather than a simple additive quality score. Combining several concerns into an invented percentage can obscure which bias mechanisms matter for the result.

The appropriate output should follow the official framework, including its signaling questions, judgment categories, and requirements for supporting explanations.

Should AI Risk-of-Bias Ratings Be Used in Evidence Synthesis?

Risk-of-bias judgments can influence sensitivity analyses, interpretation of findings, and subsequent certainty-of-evidence assessments.

An incorrect appraisal may therefore affect conclusions beyond the individual study. For example, systematically underestimating bias across included trials could lead researchers to express more confidence in an intervention effect than the evidence warrants.

AI-generated judgments should be verified before they influence the synthesis. The verification procedure should follow the applicable review standards and the official instructions for the appraisal tool.

Using AI to Support RoB 2 Assessment of an Educational Intervention

What Might AI Get Wrong?

Suppose AI assigns low risk of bias for missing outcome data because the authors state that they used multiple imputation. That conclusion would be insufficiently justified without considering the amount and pattern of missingness, the plausibility of the relevant assumptions, and the outcome-specific circumstances.

Similarly, the use of a common examination may reduce some measurement concerns but does not automatically settle every signaling question in that domain.

The example illustrates why an AI-generated appraisal should not be accepted simply because it covers every domain. Completeness of the assessment form does not establish that the judgments are methodologically defensible.

Common Misconceptions About AI Risk-of-Bias Assessment

Let AI Organize the Evidence, but Keep Appraisal Judgments Defensible

Begin by selecting the appropriate risk-of-bias instrument and identifying the result or study characteristics to which it applies. Use the official guidance to determine which information is needed for each domain.

AI may help locate that information, but reviewers should verify the passages and independently interpret their implications.

How Should AI Risk-of-Bias Performance Be Evaluated?

Compare AI-generated assessments with independently established reference judgments, ideally across different domains, study designs, and levels of reporting completeness.

Do not rely solely on overall agreement. Examine which domains produce errors, whether AI systematically underestimates risk, and whether its supporting explanations accurately reflect the source evidence.

Where AI judgments are evaluated through manual auditing, the sample should include difficult methodological cases rather than only straightforward reports.

Document the tool, model, instructions, source materials, provisional ratings, human corrections, and final decisions. These details are important for transparent reporting of AI involvement.

Before Accepting AI-Assisted Risk-of-Bias Assessments

Frequently Asked Questions About AI Risk-of-Bias Assessment

AI Can Support Critical Appraisal, but It Cannot Substitute for Defensible Judgment

How to Cite This Guide

This guide is intended to be read, shared, and used in research, teaching, and academic work. If you draw on its ideas, explanations, or other content, please acknowledge the source by citing the guide. Doing so gives appropriate credit and helps your readers locate the original resource.