Should Researchers Avoid AI When Small Errors Could Materially Harm Participants?

Should AI Be Used When One Incorrect Decision Could Seriously Affect a Participant?

A researcher proposes using an AI system to screen participants for eligibility in a psychological intervention study. The system performs well on preliminary examples, but occasionally misinterprets statements about self-harm. Another team wants to use AI-generated recommendations to determine which participants should receive additional clinical assessment.

In both situations, most outputs may be accurate. Yet an individual mistake could have consequences that are disproportionate to the apparent frequency of error.

Is high average accuracy sufficient to justify using AI? Can a human reviewer compensate for occasional mistakes? Or should researchers avoid the system when even a small error could materially harm a participant?

The decision requires examining not only how often the system fails, but what happens when it does, whether failures can be detected, and whether the research design provides adequate protection.

Avoid AI When Consequential Errors Cannot Be Adequately Prevented or Controlled

Why Small AI Errors Can Create Serious Research Risks

Error Frequency and Error Consequences Are Different

Researchers often evaluate AI performance using aggregate measures such as accuracy, sensitivity, specificity, precision, or error rates. These indicators can be useful, but they do not fully describe the consequences of a system's mistakes.

A model that correctly classifies 98% of observations may still be unsuitable for a task if the remaining errors disproportionately affect participants who require urgent protection.

The relevant concern is not simply the number of incorrect outputs. It is the relationship between those errors and the potential harm they may produce.

For example, misclassifying a participant's preferred learning format may have limited consequences in an exploratory educational study. Incorrectly classifying a participant as not requiring urgent clinical assessment could have much more serious implications.

Researchers should therefore examine error types, consequences, affected populations, and opportunities for detecting mistakes before they influence participant outcomes.

What Counts as Material Harm to a Research Participant?

Research-related harm is not limited to physical injury.

Depending on the study, AI errors may create psychological distress, privacy violations, reputational damage, economic disadvantage, discriminatory treatment, or inappropriate exclusion from beneficial research procedures.

In educational research, an incorrect AI-generated risk classification might lead to unnecessary intervention, stigmatization, or inappropriate decisions about a student's participation.

In clinical research, a mistaken recommendation could influence monitoring, referral, or treatment-related decisions if the protocol allows AI output to affect those activities.

Researchers should assess harm within the actual research context, including the vulnerability of participants and the reversibility of potential consequences.

Why Average Accuracy Can Conceal Important Failures

Aggregate performance may obscure differences across participant groups, uncommon situations, or particularly consequential error categories.

Suppose a model is evaluated using a dataset in which most participants do not require urgent intervention. A system that frequently predicts the majority category may achieve apparently high accuracy while failing to identify many participants who need additional assessment.

In such circumstances, sensitivity, false-negative rates, and performance for relevant subgroups may be more informative than overall accuracy alone.

Researchers should also examine whether the validation data adequately represent the population and circumstances in which the system will be used.

Performance estimates derived from one institution, language group, or population may not generalize reliably to another.

How Should Researchers Evaluate the Severity of Different AI Errors?

Risk assessment should consider both the probability of error and its potential consequences. The NIST AI Risk Management Framework emphasizes identifying and managing risks within the intended context of use.

For participant-facing research, an error assessment should distinguish mistakes that cause minor inconvenience from those that could expose participants to substantial harm.

These examples illustrate possible consequences rather than establish that a particular AI system will produce them. Actual risk depends on the system, population, workflow, and safeguards.

Why False Negatives and False Positives May Have Different Consequences

Classification errors are not necessarily symmetrical.

A false negative occurs when a system fails to identify a condition or characteristic that is actually present. A false positive occurs when it incorrectly identifies something as present.

In a participant safety screening task, a false negative may result in a missed referral. A false positive may cause unnecessary assessment, anxiety, or intervention.

Both can matter, but their consequences may differ substantially.

Researchers should define which errors are most consequential for the intended task and evaluate performance accordingly.

Threshold selection should be justified by the research context and applicable safety requirements, not chosen solely to maximize an aggregate metric.

Why High Performance on a Test Dataset May Be Insufficient

AI systems may perform differently when deployed outside the conditions represented in their validation data.

Changes in participant characteristics, language, institutional practices, measurement procedures, or data quality can affect performance.

A system evaluated on carefully curated data may encounter incomplete, ambiguous, or unexpected inputs during actual research operations.

Researchers should therefore assess whether the validation setting resembles the intended deployment context and whether additional local evaluation is necessary.

For consequential tasks, monitoring after deployment may also be needed to detect changes in performance or unexpected failure patterns.

Validation is not a one-time guarantee of safety under every future condition.

When Is Human Oversight Actually Protective?

Human oversight can reduce risk when reviewers have sufficient expertise, access to relevant information, and genuine authority to intervene.

However, a nominal human approval step may offer little protection if reviewers routinely accept AI recommendations without examining the evidence.

Automation bias can occur when people place excessive weight on automated recommendations, particularly when the system usually performs well or presents its conclusions confidently.

Meaningful oversight requires a workflow in which human reviewers can independently assess consequential cases, recognize uncertainty, and reject inappropriate outputs.

For high-risk decisions, the research protocol may need to ensure that qualified personnel make the final determination using validated procedures rather than relying on AI-generated advice alone.

What Safeguards Can Reduce Participant Harm?

Appropriate safeguards depend on the task and the severity of foreseeable errors.

Possible measures include restricting AI to advisory functions, requiring independent assessment before consequential action, using validated decision thresholds, monitoring subgroup performance, and establishing escalation procedures for uncertain cases.

Researchers may also need to define situations in which the AI system must not provide a recommendation or in which the workflow automatically transfers responsibility to qualified personnel.

For example, a system used to organize participant screening information might be prohibited from making final eligibility decisions. A qualified researcher could instead review the original records and apply the approved eligibility criteria.

Safeguards should be tested rather than assumed effective simply because they appear in a protocol.

When Does Ethics Review Become Necessary?

AI use that materially affects participant recruitment, eligibility, intervention allocation, safety monitoring, or other consequential research procedures may require ethics review or an amendment to an existing approval.

The applicable requirements depend on the jurisdiction, institution, research design, and nature of the AI contribution.

Under U.S. human-subjects research regulations, institutional review boards assess whether risks are minimized and reasonable in relation to anticipated benefits and the importance of the knowledge expected from the research. These requirements are described in 45 CFR 46.111 for research within its scope.

Other jurisdictions and institutions apply their own research ethics frameworks.

Researchers should not assume that approval of the original study automatically authorizes a later AI-assisted decision procedure that changes participant risks.

What if AI Creates a New Risk During an Ongoing Study?

Unexpected AI behavior may reveal a risk not adequately anticipated in the original protocol.

For example, a screening model may repeatedly misclassify participants with a particular linguistic characteristic, creating an unrecognized possibility of exclusion or delayed referral.

The research team should assess whether continued use is appropriate, implement necessary protective measures, and follow the institution's applicable reporting procedures.

For research governed by relevant U.S. Department of Health and Human Services requirements, OHRP guidance explains how unanticipated problems involving risks to participants or others should be evaluated and reported.

Not every AI error is automatically a reportable unanticipated problem. The classification depends on the event's characteristics, relationship to the research, and implications for previously recognized risks.

Where an immediate hazard exists, protecting participants takes priority over continuing the original workflow.

When Should Researchers Avoid AI Entirely for a Participant-Facing Task?

AI should not be used for a consequential participant-facing function when the research team cannot establish adequate reliability, authorization, and safeguards for the intended context.

Particularly strong reasons to avoid the system include inability to detect dangerous errors, insufficient evidence about performance in the target population, lack of qualified oversight, and foreseeable harms that cannot be reduced to an acceptable level.

Researchers should also reconsider AI when a validated non-AI procedure can achieve the research objective with fewer or more manageable risks.

The appropriate decision is not necessarily to abandon AI throughout the study. The system may remain useful for administrative or exploratory tasks that do not influence participant safety.

What should be avoided is an unjustified transfer of consequential decisions to a system whose failures cannot be adequately controlled.

Why High AI Accuracy May Still Be Inadequate for Participant Safety Screening

The 96% accuracy indicates that the classifier matched the reference classification in 960 of the 1,000 hypothetical cases. It does not establish the system's sensitivity for safety-relevant cases, its performance in the intended participant population, or the acceptability of the 12 missed flags.

The example illustrates why an apparently strong aggregate performance result may be insufficient for a high-consequence task.

The appropriate decision depends on the consequences of missed cases and the effectiveness of available safeguards, not simply the percentage of correct classifications.

Misconceptions About AI Accuracy and Participant Safety

Protect Participants Before Optimizing AI Performance or Research Efficiency

When AI may affect participant welfare, begin by identifying the decisions it would influence and the consequences of incorrect outputs.

Do not rely exclusively on aggregate accuracy. Examine consequential error categories, performance in the intended population, and whether safeguards can prevent mistakes from reaching participants.

If you cannot independently verify consequential AI outputs, that limitation becomes especially important when participants could be harmed.

Where AI requires sensitive participant information, assess whether confidentiality can be adequately protected.

The broader question of when researchers should avoid AI includes scientific validity and ethical authorization as well as participant safety.

Researchers planning participant-facing AI workflows should also determine whether the proposed AI use requires ethics review before implementation.

A defensible research protocol does not treat participant protection as a secondary consideration after model performance has been optimized. The potential consequences of failure should shape the decision to use AI from the outset.

Before Using AI for Participant-Affecting Research Decisions

Frequently Asked Questions About AI Errors and Participant Harm

Consequential Participant Risks Must Determine Whether AI Is Appropriate

How to Cite This Guide

This guide is intended to be read, shared, and used in research, teaching, and academic work. If you draw on its ideas, explanations, or other content, please acknowledge the source by citing the guide. Doing so gives appropriate credit and helps your readers locate the original resource.