Should AI Be Used to Make Clinical Research Decisions?

Should an AI System Help Decide What Happens to a Clinical Research Participant?

A clinical research team is conducting a study involving patients with a chronic medical condition. The investigators use AI to organize study documentation, summarize published evidence, and assist with administrative tasks.

The team then considers a more consequential application. Could the same technology determine whether a participant satisfies the study's eligibility criteria, identify a potential adverse event, or recommend changes to participant monitoring?

These activities differ substantially from preparing documents or summarizing literature. An incorrect recommendation may affect participant welfare, protocol compliance, or the scientific validity of the study.

The question is therefore not simply whether AI can produce clinically plausible recommendations. It is whether AI in clinical research decisions can be used with sufficient evidence, authorization, and human oversight to justify its role in consequential research activities.

Can AI Appropriately Support Clinical Research Decisions?

What Researchers Must Consider Before Using AI for Clinical Decisions

Clinical Research Decisions Are Not All Equivalent

Clinical research involves decisions ranging from routine administrative activities to actions that may directly affect participants.

AI may help organize appointments, explain technical terminology, prepare nonconfidential documents, or identify potentially relevant publications.

Other activities involve participant eligibility, treatment allocation, safety monitoring, protocol deviations, and decisions about continuing or stopping study procedures.

These tasks should not be governed by identical assumptions about acceptable AI use.

For example, an inaccurate suggested heading in a study report can usually be corrected before publication. An incorrect recommendation concerning a serious adverse event could have more immediate consequences.

The relevant assessment should therefore begin with the particular decision rather than the general label of clinical research.

Clinical Decision Support Is Different From Autonomous Clinical Decision-Making

AI can be used to provide information that supports decisions made by qualified personnel.

For example, a validated system might identify records requiring closer review or organize information relevant to an authorized clinical assessment.

This differs from allowing the system to make and implement a decision without appropriate human evaluation.

The distinction is operational rather than merely terminological. A workflow described as human supervised may still function almost autonomously if reviewers routinely accept recommendations without examining their basis.

Conversely, automated processing may be appropriate for narrowly defined tasks when the system has been adequately validated and authorized.

Researchers should evaluate the actual distribution of decision authority.

Why Clinical Plausibility Is Not Evidence of Clinical Validity

Generative AI can produce fluent explanations using medical terminology and apparently coherent clinical reasoning.

However, a plausible response may omit contraindications, misinterpret clinical information, or recommend an action inconsistent with the approved protocol.

Clinical validity requires evidence that the system performs adequately for its intended use.

This evidence may involve diagnostic or predictive performance, calibration, reliability across relevant populations, and assessment of consequential errors.

The appropriate measures depend on the task.

For example, a system designed to flag potential adverse events should be evaluated differently from one intended to summarize clinical research documentation.

Researchers should not infer clinical suitability from general-purpose benchmarks or the apparent sophistication of generated explanations.

What Does Clinical AI Validation Require?

Validation should examine whether an AI system performs its intended function under conditions relevant to the proposed clinical research workflow.

Depending on the application, this may require evaluation using representative data, independent testing, external validation, subgroup analysis, and prospective assessment.

For predictive models, discrimination and calibration may both be relevant. A model can distinguish higher-risk from lower-risk individuals while producing poorly calibrated risk estimates.

For generative systems, factual accuracy, omission of critical information, consistency, and the consequences of incorrect recommendations may require specific evaluation.

Validation should also account for the possibility that system performance changes when the population, workflow, or underlying model changes.

Reporting guidance such as CONSORT-AI and SPIRIT-AI provides relevant considerations for clinical trials involving AI interventions.

These extensions improve reporting and protocol transparency. Compliance with reporting guidance does not itself establish clinical safety or regulatory authorization.

Why the Approved Research Protocol Matters

Clinical research activities should follow the approved protocol and applicable institutional and regulatory requirements.

An AI-generated recommendation does not independently authorize a change to eligibility criteria, participant monitoring, treatment allocation, or study procedures.

For example, a system might recommend excluding a participant because of a laboratory result. Whether exclusion is appropriate depends on the approved criteria, the participant's circumstances, and the responsibilities assigned to authorized personnel.

If researchers propose introducing AI into a workflow that was not included in the approved protocol, they should determine whether an amendment or additional review is necessary.

The relevant requirements depend on the study and jurisdiction.

Researchers should consult the responsible ethics committee, sponsor, regulatory personnel, or other institutional authority rather than relying on AI to interpret its own authorization.

Participant Safety Requires Particular Attention

Participant welfare is a central concern in clinical research.

The World Medical Association's 2024 Declaration of Helsinki emphasizes the rights, safety, and well-being of research participants.

AI-assisted decisions affecting participants must be considered within these ethical responsibilities.

For example, a system that fails to recognize a serious safety concern may delay appropriate assessment. Conversely, an excessive number of incorrect alerts may create unnecessary interventions or contribute to alert fatigue.

Researchers should examine both false-negative and false-positive consequences.

Overall accuracy alone may conceal errors that are particularly consequential for participant protection.

The question of using AI to flag participant safety risks therefore requires task-specific evaluation and appropriately authorized escalation procedures.

When Does AI Become a Regulated Medical Device?

Some AI systems used in clinical contexts may fall within medical-device or clinical decision-support regulatory frameworks.

However, not every AI application used by a clinical researcher is automatically a regulated medical device.

Regulatory classification depends on factors such as intended use, functionality, claims, and applicable jurisdictional definitions.

In the United States, the Food and Drug Administration provides guidance concerning clinical decision-support software and the circumstances under which software functions may be subject to device regulation.

The European Union's AI Act also establishes obligations for defined categories of high-risk AI systems, including certain systems associated with regulated products.

Researchers should not assume that compliance with a voluntary AI governance framework replaces applicable medical-device requirements.

Regulatory classification and authorization should be determined through qualified institutional or regulatory review.

Why Human Oversight Must Be Clinically Meaningful

A clinician or investigator approving an AI recommendation does not automatically make the process safe.

The reviewer must have sufficient information and expertise to evaluate the recommendation and authority to reject or modify it.

For example, an AI-generated eligibility recommendation may omit a relevant exclusion criterion.

If the reviewer examines only the system's final classification, the omission may remain undetected.

Meaningful review may require access to the relevant participant information, the approved protocol, and the basis for the recommendation.

The appropriate oversight arrangement depends on the task and potential consequences.

This is why genuine human oversight must be designed around the actual decision rather than a generic approval requirement.

Automation Bias Can Affect Clinical Research Judgment

Automation bias occurs when people give automated recommendations inappropriate priority over other relevant evidence.

Parasuraman and Manzey (2010) examine how automation complacency and attentional processes may affect human decision-making.

In clinical research, a reviewer may become accustomed to accurate AI recommendations and gradually reduce independent checking.

This could be particularly concerning when an incorrect recommendation appears technically credible.

However, the presence of AI does not establish that automation bias will occur in every clinical workflow.

Researchers should evaluate the design of the review process, including whether reviewers can identify relevant errors and whether disagreement with AI is operationally supported.

Data Privacy Is Part of Clinical AI Safety

Clinical research data may include identifiable health information, genetic information, medical histories, and other sensitive records.

Using an external AI service may introduce additional processing arrangements involving providers or subprocessors.

Researchers should determine whether the proposed processing is authorized under participant consent, institutional agreements, applicable data-protection law, and the approved protocol.

Removing direct identifiers does not necessarily eliminate reidentification risk.

Provider terms concerning retention, secondary use, security, and data transfer should be evaluated before information is submitted.

A system can generate accurate clinical recommendations while still being unsuitable because its processing arrangements violate confidentiality obligations.

What About AI-Assisted Participant Eligibility Screening?

Eligibility screening illustrates why clinical AI applications require careful distinction between support and final decisions.

An AI system may help identify records containing information relevant to predefined inclusion or exclusion criteria.

However, eligibility may depend on clinical interpretation, incomplete documentation, protocol-specific definitions, or information requiring clarification.

A system that overlooks these considerations could incorrectly include or exclude participants.

Researchers should therefore assess AI-assisted participant eligibility separately from routine administrative screening.

Any final eligibility determination should follow the approved protocol and applicable authorization requirements.

Clinical Research AI Requires Clear Accountability

Research teams should identify who is responsible for validating the system, reviewing recommendations, implementing decisions, and responding to errors.

Responsibilities may be distributed among investigators, clinicians, statisticians, data managers, and institutional personnel.

However, assigning an AI system a task does not transfer scholarly or professional accountability to the technology.

The International Committee of Medical Journal Editors emphasizes human responsibility for the accuracy and integrity of AI-assisted scholarly work.

Clinical and regulatory responsibilities may impose additional obligations beyond publication authorship.

Researchers should therefore distinguish scientific accountability, clinical decision authority, and regulatory responsibility.

When Should AI Not Be Used for a Clinical Decision?

AI may be inappropriate when its performance cannot be adequately validated, when relevant errors cannot be detected, or when the proposed workflow lacks sufficient human oversight.

It may also be unsuitable when the required data processing is unauthorized or when the system's intended use conflicts with the approved protocol.

For example, an unvalidated general-purpose language model should not be treated as an independent authority for making participant safety decisions.

Additional prompting does not establish the validation required for a consequential clinical application.

Researchers should consider alternative procedures when adequate safeguards cannot be demonstrated.

The objective is not to reject all clinical AI applications but to ensure that their use is justified by appropriate evidence and governance.

Evaluating AI Assistance for Clinical Trial Eligibility Decisions

The example does not establish that the proposed AI system is suitable for clinical screening.

Its acceptability would depend on actual validation evidence, the approved protocol, and applicable requirements.

The central distinction is that AI may support an authorized process without automatically acquiring authority over consequential participant-related decisions.

Misconceptions About AI in Clinical Research Decisions

How to Decide Whether AI Should Support a Clinical Research Decision

Researchers should evaluate the intended decision before selecting an AI system.

The assessment should establish the clinical or scientific purpose, potential consequences, available validation evidence, and applicable authorization requirements.

Define the Intended Use

Specify what information AI will process, what output it will generate, and how that output will influence a clinical research decision.

Descriptions such as "AI-assisted clinical analysis" are insufficiently precise for meaningful risk assessment.

Require Appropriate Validation

Evaluate performance using evidence relevant to the population, task, and workflow.

Pay attention to consequential error types rather than relying exclusively on overall accuracy.

Preserve Authorized Clinical Judgment

Ensure that decisions requiring qualified clinical or research personnel remain within their authority.

Reviewers should be able to inspect the evidence, challenge recommendations, and prevent inappropriate actions.

Review Ethical and Regulatory Obligations

Consult the responsible institutional authorities when AI use may affect approved procedures, participant rights, data processing, or regulated software functions.

Do not treat AI-generated interpretations of regulatory requirements as official authorization.

Establish Monitoring and Corrective Procedures

For authorized consequential applications, identify how errors will be detected, reported, investigated, and corrected.

Reassess system performance when relevant data, workflows, or software versions change.

Before Using AI for Clinical Research Decisions

Common Questions About AI in Clinical Research

Clinical AI Must Be Justified by Evidence and Accountable Oversight

How to Cite This Guide

This guide is intended to be read, shared, and used in research, teaching, and academic work. If you draw on its ideas, explanations, or other content, please acknowledge the source by citing the guide. Doing so gives appropriate credit and helps your readers locate the original resource.