03 · What You Need to Know
How to Evaluate Participant Harm and Scientific Consequences of AI Errors
Why the Consequences of an Error Matter More Than Its Appearance
AI-generated errors can appear in ordinary text, statistical code, classification labels, or automated recommendations.
The seriousness of an error depends less on its format than on how the output is used.
For example, an incorrect numerical value in an exploratory draft may be corrected before it influences any decision. The same value, if incorporated into a published table or used to determine participant eligibility, could have substantially greater consequences.
Researchers should therefore examine the downstream role of each AI output.
An apparently minor error can become consequential when it propagates through subsequent analytical steps.
What Counts as Participant Harm?
Participant harm may involve physical, psychological, social, economic, privacy-related, or other adverse consequences.
The relevant harms depend on the study population, intervention, information collected, and research context.
For example, an AI system used to screen participants for a sensitive intervention may incorrectly classify someone as eligible.
An AI-generated summary might also reveal confidential information or misrepresent a participant's experiences in ways that create reputational consequences.
Researchers should consider foreseeable harms associated with the actual procedure rather than assume that all AI research risks concern technical accuracy alone.
What Makes an AI Error Scientifically Material?
An error is scientifically material when it could meaningfully affect the evidence, analysis, interpretation, or conclusions of the study.
For example, incorrectly reversing a questionnaire item may change composite scores and the estimated relationship between variables.
An AI-generated literature summary that misrepresents a central publication may distort the theoretical argument.
A classification error may affect estimates of category prevalence or differences between participant groups.
Materiality should be assessed in relation to the study's research questions and claims, not merely whether a numerical value changes.
A small numerical discrepancy can matter when it changes an important inference, while a larger difference may be inconsequential for another purpose.
Which Dimensions Should Researchers Examine?
| Dimension |
Assessment question |
Illustrative concern |
| Severity |
How serious could the consequences be? |
Participant exposure to an unsuitable intervention |
| Likelihood and uncertainty |
What evidence exists about possible error frequency? |
Unvalidated classification of multilingual responses |
| Detectability |
Can mistakes be recognized before they matter? |
Plausible but incorrect statistical transformation |
| Reversibility |
Can the consequences be corrected? |
Manuscript wording error versus unauthorized disclosure |
| Propagation |
Can one error affect subsequent operations? |
Incorrect variable coding used across several analyses |
| Exposure |
How many people or records could be affected? |
Automated decisions applied to an entire participant cohort |
| Safeguards |
Can effective controls reduce the risk? |
Qualified review before consequential decisions |
These dimensions form a practical assessment framework, not a validated universal risk score. Researchers should avoid treating subjective categories as precise numerical measurements without appropriate justification.
Why Low-Probability Errors Can Still Matter
An AI system may perform well on average while producing occasional errors with serious consequences.
For example, an automated eligibility system may classify most participants correctly but fail on a rare condition that is important for safety.
High overall accuracy does not necessarily establish acceptable performance for every consequential category.
Researchers should examine error types and the consequences associated with them.
Where potential harm is substantial, limited evidence about rare failures may justify additional testing or a more conservative workflow.
How Can AI Errors Affect Research Measurements?
AI-generated classifications, scores, and categories may become variables in a research dataset.
Errors in these outputs can introduce measurement error or systematic bias.
For example, a classifier may interpret cautious student responses as positive engagement more frequently than negative engagement.
If the resulting labels are used to compare academic programs, the observed differences may partly reflect classification behavior rather than actual differences between students.
Researchers should evaluate measurement validity and relevant error patterns before using AI-generated variables in consequential analyses.
Why Systematic Errors May Be More Serious Than Random Mistakes
Random errors may increase uncertainty, although their consequences depend on the analytical setting.
Systematic errors can shift measurements or estimates in a particular direction.
For example, an AI classifier that consistently underrecognizes critical responses from one linguistic group may distort comparisons between groups.
Researchers should therefore examine whether errors are associated with particular categories, populations, or contexts.
Overall performance measures may conceal such patterns.
Task-specific validation should reflect the population and decisions for which the system will be used.
How Can AI Errors Propagate Through an Analysis?
An error introduced early in the workflow may influence several downstream results.
For example, an AI-generated script may incorrectly exclude valid survey responses.
The resulting dataset may then produce altered descriptive statistics, regression estimates, figures, and manuscript interpretations.
Because the outputs may remain internally consistent, ordinary proofreading may not reveal the original mistake.
Researchers should identify critical points where errors can propagate and verify those operations before subsequent analyses depend on them.
Can an AI Error Affect Participants Without Changing the Published Findings?
Yes. Participant welfare and scientific validity are separate concerns.
An AI system might disclose confidential information while generating analytically accurate results.
Similarly, an automated participant-facing message may cause distress even when the study's aggregate findings remain unchanged.
Researchers should therefore evaluate participant risks independently of the accuracy of the final scientific conclusions.
A technically correct output does not establish that the processing or intervention was ethically acceptable.
Can an AI Error Change Scientific Conclusions Without Harming Participants Directly?
Yes. An AI-generated analytical error may compromise the published evidence without producing immediate participant harm.
For example, incorrect statistical code may produce a misleading association that influences later research or policy discussions.
The consequences may extend beyond the original dataset.
Researchers should assess scientific integrity even when the study involves public or non-identifiable information.
Participant protection and research validity both matter, but they require different forms of assurance.
Why Detectability Should Influence the Decision
Some errors are readily recognized through ordinary review.
Others may remain hidden because the output is plausible or because the reviewer lacks the necessary expertise.
For example, a generated citation may look authentic until the original publication is searched.
A statistical script may run without warnings while using an incorrect denominator.
Researchers should establish whether suitable verification procedures can identify consequential errors before the outputs influence decisions or conclusions.
When errors cannot be adequately detected, the proposed AI use may be inappropriate.
What Does Reversibility Mean?
Reversibility concerns whether an error and its consequences can be corrected after discovery.
An incorrect manuscript heading can usually be revised without substantial scientific consequences.
An unauthorized transfer of confidential participant information may be difficult or impossible to reverse fully.
An incorrect research finding may require reanalysis and correction of the scholarly record.
Researchers should distinguish correcting an output from reversing all consequences that occurred because of it.
Where consequences are difficult to reverse, preventive safeguards become particularly important.
How Should Researchers Evaluate AI-Assisted Participant Screening?
Participant screening may influence eligibility, exposure to interventions, or access to research opportunities.
Researchers should establish the approved criteria and determine whether the AI system can apply them reliably.
Validation should consider false inclusions, false exclusions, and their respective consequences.
Qualified human oversight may be required for consequential decisions under the applicable research protocol.
However, human review should not be treated as an effective safeguard unless reviewers have the expertise and information needed to identify mistakes.
If acceptable safety conditions cannot be established, automated screening should not proceed.
How Should Researchers Evaluate AI-Generated Statistical Results?
Researchers should distinguish computational errors from methodological errors.
Computational errors arise when code or calculations incorrectly implement an intended operation.
Methodological errors arise when the operation itself is inappropriate for the research question or data.
For example, AI-generated code may correctly calculate a regression coefficient while ignoring important clustering in the study design.
Independent verification should examine both implementation and methodological suitability.
Correct execution alone does not establish scientific validity.
When Should Researchers Use Sensitivity Analysis?
Sensitivity analysis can help determine whether conclusions depend on uncertain analytical choices or potentially incorrect AI outputs.
For example, researchers may compare findings under alternative plausible classifications or data-processing assumptions.
If the conclusions change substantially, the uncertainty may be scientifically important.
However, sensitivity analysis does not automatically validate the original AI procedure.
It provides evidence about how particular assumptions or errors may affect the findings.
The selected analyses should be justified and reported accurately.
How Should Researchers Choose Safeguards?
Safeguards should address the actual failure modes of the task.
For literature claims, original-source verification may be appropriate.
For statistical code, independent calculations and methodological review may be necessary.
For AI-generated classifications, task-specific validation and review of consequential errors may be required.
For participant-facing recommendations, approved procedures and qualified oversight may be essential.
Researchers should evaluate whether the safeguards are effective rather than simply count how many controls have been introduced.
What Is Residual Risk?
Residual risk is the risk that remains after safeguards have been applied.
For example, independent review may reduce the likelihood of undetected analytical errors but cannot guarantee that every mistake will be found.
A validated classifier may still perform poorly on unusual cases.
Researchers should determine whether the remaining uncertainty and consequences are acceptable for the intended use.
The existence of safeguards does not automatically establish that residual risk is acceptable.
When Should Researchers Reject the AI Procedure?
Researchers should avoid an AI procedure when foreseeable consequential errors cannot be adequately detected or controlled.
They should also avoid processing arrangements that violate applicable confidentiality, ethics, or institutional requirements.
The appropriate alternative may involve a validated non-AI method, a redesigned task, or qualified human assessment.
Rejecting a particular AI application does not imply that the technology is unsuitable for every research purpose.
The decision concerns whether the proposed use can satisfy the requirements of the study.
