03 · What You Need to Know
How to Evaluate AI When Participant Harm Is a Foreseeable Consequence
What Counts as Material Harm to a Research Participant?
Material harm refers here to an adverse consequence sufficiently consequential to affect a participant's safety, rights, welfare, privacy, or important interests. It is a practical risk-assessment concept rather than a single legal category with a universally applicable definition.
Harm may be physical, psychological, social, economic, or informational. Its significance depends on the participant's circumstances and the context in which an AI output is used.
For example, an incorrect transcription of a non-sensitive interview passage may be corrected during routine quality assurance. An incorrect transcription that changes the meaning of a disclosure about imminent danger could have substantially greater consequences if it influences a research team's response.
Materiality therefore concerns more than whether an AI output is inaccurate. Researchers must consider what decisions depend on the output and what may happen when those decisions are wrong.
Distinguish Analytical Errors From Errors That Affect Participants Directly
Not all research errors create the same kind of risk.
| Type of error |
Illustrative consequence |
Primary concern |
| Incorrect preliminary thematic code |
A participant statement is assigned to an unsuitable category |
Validity of analysis and potential misrepresentation |
| Incorrect sensitive classification |
A participant is wrongly described as having a stigmatizing characteristic |
Privacy, dignity, and reputational harm |
| Incorrect eligibility determination |
An ineligible participant enters a procedure or an eligible participant is excluded |
Participant protection, fairness, and protocol compliance |
| Missed safety warning |
A potentially serious adverse event is not escalated |
Delayed assessment or protective action |
| Incorrect intervention recommendation |
An inappropriate action is taken based on an AI-generated assessment |
Direct physical or psychological harm |
These examples are illustrative. An error that appears minor in isolation may become consequential if it affects subsequent decisions or is repeatedly reproduced across a research workflow.
Researchers should map the complete sequence from AI input to output, human interpretation, decision, and possible participant consequence.
Risk Depends on Both the Likelihood and Severity of Harm
A useful assessment considers how frequently errors might occur and how serious their consequences could be. However, a simple numerical risk score may conceal important differences between scenarios.
For example, a relatively common but easily corrected formatting error may be less concerning than a rare error that could lead to irreversible injury.
Researchers should examine the severity of potential harm, its likelihood under realistic operating conditions, the number of participants exposed, and whether an error can be detected before it affects someone.
They should also consider whether harm is reversible, whether affected participants have access to an effective remedy, and whether particular groups face disproportionate risks.
The distinction between ordinary and high-stakes AI research tasks is therefore determined by the consequences of using the output, not merely the technical complexity of the model.
False Positives and False Negatives Can Produce Different Harms
Many AI classification systems produce two particularly important types of error.
False positive
The system identifies a condition, characteristic, or risk that is not present according to the applicable reference standard.
False negative
The system fails to identify a condition, characteristic, or risk that is present according to the applicable reference standard.
The consequences are not necessarily symmetrical. In participant safety screening, a false negative might delay an appropriate response. A false positive might trigger unnecessary procedures, distress, or disclosure of sensitive information.
Researchers should examine sensitivity and specificity where appropriate, but these measures alone do not establish whether a tool is safe for the intended purpose. Positive and negative predictive values also depend on the prevalence of the condition in the population being assessed.
Decision thresholds should reflect the actual research context, including the relative consequences of different errors. A threshold selected because it maximizes overall accuracy may not minimize participant harm.
Validation Must Reflect the Intended Participants and Conditions
A system that performs well in a published benchmark may not perform equally well in a different research setting.
Differences in participant age, language, health status, cultural context, data quality, or the prevalence of relevant conditions can affect performance. Model updates and changes in how researchers use the system may introduce further uncertainty.
For consequential applications, validation should address the intended task and population rather than rely exclusively on general-purpose performance claims.
Depending on the application, appropriate evaluation may include comparison with a defensible reference standard, subgroup analysis, assessment of calibration, examination of uncertainty, and testing under realistic failure conditions.
Researchers should also determine whether the evidence comes from independent evaluation or only from the system's developer.
Importantly, validation does not eliminate error. It provides evidence about the conditions under which the system may be used and the limitations that remain.
Human Oversight Is Not the Same as Human Protection
Adding a human reviewer does not automatically prevent harmful AI errors.
Reviewers may become overly reliant on AI suggestions, particularly when outputs are fluent, detailed, or presented with apparent confidence. They may also lack sufficient access to the original information needed to recognize an error.
Meaningful oversight requires reviewers with appropriate expertise, sufficient time, access to independent evidence, and authority to reject the AI output.
The review procedure should identify which outputs require verification, what evidence reviewers must examine, how disagreements are resolved, and what happens when the system produces uncertain or contradictory results.
For particularly consequential decisions, researchers may need independent verification by more than one qualified reviewer. Whether that safeguard is necessary depends on the risk and the effectiveness of other controls.
Some Decisions Should Not Depend on Autonomous AI Outputs
Where an AI system could directly influence participant safety, researchers should distinguish analytical assistance from decision authority.
AI may help organize information for review without being authorized to determine the final action. This distinction matters when systems are proposed for assessing research participant eligibility, flagging participant safety risks, or supporting clinical research decisions.
Autonomous use may be inappropriate when the system lacks adequate validation, cannot reliably identify uncertainty, or produces errors that are difficult to detect before harm occurs.
Researchers should establish explicit boundaries around what the AI system may recommend, what it may initiate, and which actions require qualified human authorization.
Ethics Review Should Address AI-Specific Failure Scenarios
Research ethics committees assess proposed risks and protections within their applicable regulatory and institutional frameworks. Where AI is introduced into participant-facing procedures, the review should consider its actual function and foreseeable consequences.
A protocol may need to describe the model or system, the evidence supporting its use, its limitations, the data it processes, and the human oversight arrangements.
Researchers should also establish procedures for adverse events, unexpected model behavior, privacy incidents, and circumstances requiring suspension of AI-assisted activities.
Under the World Medical Association's Declaration of Helsinki, medical research involving human participants must prioritize their rights and interests and include appropriate risk assessment and management. Other research domains may be governed by different frameworks, but the underlying responsibility to protect participants remains central.
Researchers Need Clear Stopping and Escalation Rules
A risk-management plan should specify when an AI-assisted process must be paused, reviewed, or discontinued.
Possible triggers include a serious unexpected error, evidence of unacceptable subgroup performance, a confidentiality incident, or a model change that invalidates previous evaluation.
Escalation procedures should identify who receives reports, who has authority to suspend the workflow, and how affected participants will be protected.
Watch Out
Do not continue using a consequential AI workflow solely because no harm has yet been reported. The absence of observed incidents may reflect limited monitoring, small sample sizes, or failures that have not been detected.
Researchers should distinguish temporary suspension for investigation from permanent discontinuation. The appropriate response depends on the severity of the concern, available evidence, and whether the risk can be adequately addressed.
04 · A Practical Example
Evaluating an AI System That Flags Possible Participant Safety Concerns
Hypothetical Example
An AI-Assisted Safety Review in a Longitudinal Study
A research team conducts a longitudinal study involving 500 adult participants. Participants periodically complete questionnaires that include questions about psychological distress.
The team considers an AI system that would flag responses for expedited review by trained personnel. The proposed system would not provide diagnoses or independently determine interventions.
During preliminary validation against a qualified reference assessment, the system is evaluated on 100 illustrative cases. Of 20 cases requiring expedited review, it correctly flags 16 and misses 4. Among the remaining 80 cases, it incorrectly flags 12.
In this hypothetical sample, the system identifies 80% of cases requiring expedited review, while missing 20%. Its specificity is 68/80, or 85%. These figures describe the illustrative evaluation sample and should not be interpreted as evidence of real-world performance.
Whether such performance is acceptable depends on the seriousness of missed cases, the reliability of the reference assessment, and the availability of additional protective procedures.
Step 1: Identify the consequence of missed cases.
The team examines whether false negatives could delay important safety responses and whether an independent review procedure would still identify participants needing attention.
Step 2: Examine the false-positive burden.
Researchers assess whether unnecessary flags could create distress, compromise confidentiality, or place excessive demands on qualified reviewers.
Step 3: Evaluate the intended population.
The team investigates whether the validation sample adequately represents the participants and conditions expected during the study.
Step 4: Define human decision authority.
Qualified personnel retain responsibility for interpreting flagged responses and determining appropriate actions under the approved safety protocol.
Step 5: Decide whether deployment is justified.
If the system cannot demonstrate an acceptable risk profile, the researchers do not deploy it. They may instead use established questionnaire thresholds and direct human review.
The example illustrates why an apparently respectable performance percentage cannot establish ethical acceptability. The consequences of the missed cases may be more important than the overall proportion of correct classifications.
