03 · What You Need to Know
Recognizing When AI Should Not Be Used in a Research Workflow
Serious Consequences Change the Burden of Justification
Research methods should be selected according to their suitability for the research question, not simply because a technology is available.
When AI is used for routine formatting or preliminary organization of non-sensitive information, the consequences of error may be relatively limited. An incorrect output can often be identified and corrected before it affects the research.
When AI influences participant safety, medical decisions, sensitive classifications, or the allocation of important opportunities, the consequences can be substantially different.
In these circumstances, researchers should require stronger evidence that the proposed application is appropriate and that foreseeable failures can be managed.
The burden of justification generally increases with the severity of possible harm, particularly when an error is difficult to reverse or detect.
This reflects the broader principle that AI should be used differently in high-stakes research because the consequences of its outputs differ from those of ordinary administrative or exploratory tasks.
When Errors Could Cause Irreversible or Catastrophic Harm
Some errors may produce consequences that cannot be fully corrected afterward.
Examples include inappropriate exposure to a hazardous research procedure, delayed response to a serious adverse event, irreversible disclosure of highly sensitive participant information, or an unjustified decision that substantially compromises participant welfare.
The possibility of such harm does not mean that every associated research activity must be prohibited. However, it creates a strong reason to avoid giving an inadequately evaluated AI system autonomous authority over the consequential decision.
Researchers should ask whether the error could be detected before harm occurs, whether an independent safeguard would prevent the consequence, and whether the proposed AI system offers benefits that justify the remaining risk.
Where those questions cannot be answered satisfactorily, the AI application should not proceed in its proposed form.
When the AI System Has Not Been Validated for the Intended Purpose
A model's ability to generate convincing answers does not establish that it is suitable for a consequential research task.
General benchmarks, vendor demonstrations, or positive experiences in unrelated applications may provide preliminary information. They do not necessarily demonstrate performance in the population, environment, and decision context of a particular study.
For example, an AI model evaluated on general medical questions should not automatically be considered suitable for determining whether a participant satisfies a complex clinical exclusion criterion.
Researchers should consider whether the system has been evaluated against an appropriate reference standard, whether the validation population resembles the intended participants, and whether the relevant error types have been examined.
When adequate validation is unavailable and the consequences of an incorrect decision are serious, avoiding operational use may be more defensible than relying on unverified performance assumptions.
A limited research evaluation of the AI system itself may still be possible under appropriate safeguards, provided that its outputs do not expose participants to unjustified risks.
When Reviewers Cannot Independently Verify the Output
Human oversight is frequently proposed as a solution to AI uncertainty. Its effectiveness, however, depends on what reviewers can actually evaluate.
If the AI system produces a recommendation without sufficient supporting evidence, a reviewer may be unable to determine whether the conclusion is correct.
Likewise, a reviewer who lacks the necessary professional expertise may approve an output without recognizing an important error.
In consequential research, an approval step that cannot reliably detect harmful mistakes should not be treated as an adequate safeguard.
Researchers should establish whether human verification can meaningfully control the relevant risks. If it cannot, the appropriate response may be to remove AI from the decision rather than increase the number of superficial approvals.
When Even Multiple Reviewers Cannot Resolve the Uncertainty
Independent review can reduce some errors, but it cannot compensate for missing evidence, invalid measurement criteria, or an inherently unsuitable analytical method.
Suppose an AI system attempts to infer a participant's psychiatric diagnosis from a short interview excerpt. Two reviewers might agree that the model's interpretation sounds plausible, but neither can establish the diagnosis from the available information.
Adding more reviewers would not necessarily solve the problem. The limitation lies in the evidence and the proposed inference.
Researchers should distinguish errors that can be corrected through additional review from limitations that require a different measurement procedure or research design.
Even double human verification may be insufficient when reviewers cannot independently establish the relevant facts.
When the AI System Cannot Reliably Recognize Its Own Limitations
Some AI systems produce confident answers despite incomplete information, ambiguous evidence, or unsupported assumptions.
This becomes particularly concerning when the research workflow assumes that the system will identify uncertainty and request human assistance whenever necessary.
A model that fails to recognize its own errors may not reliably trigger the very safeguard intended to prevent harm.
Researchers should not assume that an AI-generated confidence statement accurately represents the probability that a recommendation is correct. Confidence measures require appropriate definition, calibration, and validation.
Where the system cannot reliably identify cases requiring escalation, the workflow may need universal independent review, a separate validated screening procedure, or removal of AI from the consequential decision.
When Errors Are Difficult to Detect Before They Affect Participants
The timing of error detection is often as important as the estimated error rate.
An incorrect AI-generated summary may be corrected before publication. By contrast, an incorrect eligibility recommendation may influence a participant's enrollment before the mistake becomes apparent.
Researchers should consider whether the proposed workflow contains an effective checkpoint between the AI output and the action that could cause harm.
Retrospective audits may improve future performance, but they cannot necessarily reverse consequences that have already occurred.
When a harmful action may occur before an error can reasonably be detected, researchers should avoid autonomous reliance on the AI system unless other safeguards adequately prevent the consequence.
When AI Creates Unnecessary Privacy or Confidentiality Exposure
AI use may be inappropriate even when its analytical outputs are accurate.
For example, transferring confidential participant interviews to an unauthorized external service could create unacceptable privacy risks regardless of whether the resulting analysis is useful.
Researchers must examine the actual data-processing arrangements, including access, retention, security, contractual restrictions, and applicable legal requirements.
Consent or ethics approval for collecting information does not automatically authorize every subsequent processing activity.
Particular caution is warranted when AI is proposed for analyzing highly sensitive participant characteristics or research involving children and other vulnerable populations.
Where confidentiality requirements cannot be satisfied, the proposed AI processing should not proceed. An institutionally authorized alternative may be considered if it adequately addresses the relevant risks.
When the AI Task Is Scientifically Unjustified
Ethical acceptability requires more than demonstrating that a tool can perform a task.
Researchers should establish why the proposed AI analysis is necessary and whether its outputs can validly answer the research question.
For instance, using AI to infer a participant's moral character, psychological condition, or social identity from unrelated behavioral indicators may lack a defensible measurement basis.
Even if the system produces consistent classifications, those classifications may not represent the intended construct.
Scientific invalidity can itself create ethical concerns when participants are exposed to risks without a reasonable prospect of generating meaningful knowledge.
Researchers should avoid AI applications that introduce consequential judgments without an adequate theoretical, methodological, or evidentiary foundation.
When a Safer and Scientifically Adequate Alternative Exists
AI should not be evaluated in isolation from available alternatives.
A proposed AI workflow may be faster or less expensive than manual assessment, but these advantages must be weighed against the additional risks and resources required for validation and oversight.
In some cases, an established rule-based procedure, validated instrument, or qualified human assessment may provide a more defensible approach.
This does not imply that conventional methods are error-free or that human judgment is always superior to AI. The relevant comparison concerns the evidence supporting each method in the intended context.
Researchers should consider comparative performance, transparency, operational feasibility, participant burden, and the consequences of failure.
| Research situation |
Reason to reconsider AI |
Potential alternative |
| AI determines eligibility for a potentially risky intervention |
Incorrect decisions may expose participants to harm |
Protocol-based assessment by appropriately qualified personnel |
| AI infers clinical diagnoses from unstructured narratives |
Insufficient construct validity or task-specific validation |
Appropriate validated assessment and qualified clinical interpretation |
| AI processes confidential data through an unauthorized service |
Unacceptable disclosure or compliance risk |
Approved secure processing or analysis without the external service |
| AI independently determines whether a study should stop |
Consequential decision requires appropriate statistical, clinical, and governance procedures |
Prespecified monitoring rules and authorized human decision-making |
| AI generates unsupported sensitive classifications |
Potential stigma and invalid measurement |
Participant-reported measures or defensible research instruments |
These alternatives are illustrative and must themselves be evaluated for suitability. The goal is not to privilege human procedures automatically, but to choose the method with the strongest defensible balance of scientific value and participant protection.
When AI Use Conflicts With Applicable Requirements
Researchers must comply with relevant institutional policies, ethics approvals, contractual restrictions, and applicable laws or regulations.
Some activities may require specific authorization, professional supervision, or conformity with regulatory requirements before an AI system can be used operationally.
For example, AI software used for certain medical purposes may fall within medical device regulatory frameworks, depending on its intended use and jurisdiction. Research ethics approval alone does not necessarily satisfy those requirements.
Similarly, processing sensitive personal information may be subject to restrictions that cannot be overcome simply by adopting an internal verification procedure.
When the proposed AI use is prohibited or lacks required authorization, researchers should not proceed while attempting to compensate through additional human review.
When Model Changes Make Previous Safety Evidence Unreliable
An AI system may change during a study through model updates, modified prompts, altered processing pipelines, or changes in external dependencies.
Such changes may affect performance even when the product retains the same commercial name.
Researchers should determine whether the modification is material to the validated application and whether existing evidence remains applicable.
For consequential tasks, an unassessed change may justify suspending the AI-assisted workflow until its implications are evaluated.
Continued use should not depend solely on the assumption that a newer model is necessarily more accurate or safer.
When the Remaining Risk Is Unacceptable Despite Safeguards
Risk cannot always be eliminated. The relevant question is whether the remaining risk is ethically acceptable in relation to the anticipated benefits and the available alternatives.
Researchers should avoid treating a collection of safeguards as proof that a workflow is safe. Each safeguard must address a relevant failure mode and operate effectively under realistic conditions.
A validated model, human reviewer, audit trail, and escalation procedure may collectively reduce risk. However, if a serious failure remains plausible and cannot be adequately prevented or managed, the application may still be inappropriate.
Watch Out
Do not use the absence of previously reported harm as sufficient evidence that a high-stakes AI application is safe. Limited exposure, incomplete monitoring, and rare but severe failures may prevent important risks from becoming visible during early use.
Researchers should also avoid treating risk acceptance as an exclusively technical judgment. Where participants may bear substantial consequences, ethical review, relevant professional expertise, and appropriate institutional accountability are essential.
