Can an AI Interviewer Respond Inappropriately to Participant Distress?

What Happens When a Research Participant Becomes Distressed During an AI Interview?

Suppose an AI chatbot interviews university teachers about workplace experiences. A participant begins describing severe professional stress, then writes, "I don't think I can keep talking about this. It's bringing back some difficult memories."

A trained human interviewer might recognize the discomfort, stop probing, and ask whether the participant wants to end the interview. But what if the chatbot interprets the statement as an opportunity to collect more detailed information?

The concern is not simply whether AI can produce empathetic language. It is whether an automated interviewer can recognize when ordinary research questioning has become inappropriate and respond in a way that protects the participant.

Yes, AI Interviewers Can Respond Inappropriately to Distress

Why Participant Distress Creates Special Risks in AI-Mediated Interviews

What Counts as Participant Distress?

Participant distress refers to emotional or psychological discomfort experienced during research participation. It may include anxiety, sadness, embarrassment, fear, agitation, or feelings of being overwhelmed.

Distress can arise when interview questions concern difficult experiences, when participants unexpectedly recall upsetting events, or when the interviewing interaction itself becomes uncomfortable.

Not every expression of discomfort indicates an emergency. A participant may become briefly emotional while discussing a meaningful experience and still wish to continue. Others may need the interview to stop immediately.

The researcher's responsibility is to anticipate reasonably foreseeable risks and establish procedures appropriate to the study's population and subject matter.

For AI-mediated interviewing, this includes considering whether the system can recognize relevant expressions of discomfort and whether it is authorized to continue questioning.

How Might an AI Interviewer Respond Inappropriately?

Several kinds of failure are possible. They differ in severity, but each can affect participant welfare or the integrity of the research procedure.

These are foreseeable failure modes rather than evidence that every AI interviewer will exhibit them. Their likelihood and consequences depend on the system, research context, and safeguards implemented.

Why Can Empathetic AI Responses Be Misleading?

Large language models can generate language that resembles empathy, including expressions of concern, validation, and encouragement. Participants may perceive such responses as supportive.

However, producing empathetic language does not establish that the system understands the participant's psychological condition or can judge the consequences of continued questioning.

For example, a chatbot might respond to distress by saying, "I understand exactly how you feel." This could imply a degree of personal understanding that the system does not possess.

It might also offer reassuring statements without knowing whether the participant is experiencing ordinary discomfort, serious psychological distress, or a situation requiring professional assistance.

The World Health Organization's guidance on artificial intelligence for health emphasizes human autonomy, safety, accountability, and appropriate oversight. Its 2023 statement concerning large language models also warns that plausible and authoritative-sounding AI responses may contain serious errors. WHO's guidance supports caution when AI-generated responses could affect human well-being.

Although this guidance primarily addresses health-related AI, the underlying safety concerns are relevant when research chatbots encounter psychologically sensitive disclosures.

Can AI Mistake Distress for Useful Interview Data?

Yes. A system designed to obtain detailed responses may interpret an emotionally significant statement as material requiring further exploration.

Suppose a participant says, "Talking about how I was treated at work is making me uncomfortable."

A chatbot optimized for elaboration might ask, "Could you describe exactly what happened and how it affected you?"

That question may be relevant to the research topic, but relevance alone does not make it appropriate. The participant has expressed discomfort, and further probing could conflict with their wishes.

This illustrates why the rules governing AI-generated follow-up questions must include explicit conditions under which probing is prohibited.

The system should not treat the pursuit of richer data as more important than the participant's right to decline, pause, or withdraw.

Can AI Reliably Recognize Distress From Written Responses?

AI systems can identify some explicit expressions of distress, particularly statements such as "I want to stop" or "This question is upsetting me."

However, distress is not always expressed directly. Participants may use indirect language, humor, silence, unusually brief responses, or culturally specific expressions.

Text-based systems also lack access to many nonverbal signals available during face-to-face interviews. Voice-based systems may capture additional cues, but interpreting those signals reliably remains a separate technical and methodological challenge.

A chatbot may therefore miss distress, incorrectly classify an ordinary statement as alarming, or respond inconsistently to similar expressions.

Researchers should not assume that a general sentiment classifier or an LLM instruction to detect distress constitutes a validated participant-safety mechanism.

Where distress detection is used, its performance should be tested with realistic scenarios relevant to the study population and language context.

What Should an AI Interviewer Do When a Participant Expresses Distress?

The appropriate response depends on the seriousness of the situation and the study's approved safety protocol.

For ordinary discomfort, the system may need to acknowledge the participant's concern, stop the current line of questioning, and offer a choice to pause, skip the question, or end the interview.

For more serious situations, the system may need to terminate automated probing and activate a predefined human-contact procedure.

This sequence is an illustrative safeguard design, not a universally mandated protocol. The actual response should reflect the research population, foreseeable risks, institutional requirements, and available support arrangements.

When Does Distress Become an Ethical or Safety Incident?

Not every emotional response during an interview constitutes an adverse event or an unanticipated problem requiring formal reporting.

Under guidance from the U.S. Office for Human Research Protections, an unanticipated problem generally involves an event that is unexpected, related or possibly related to research participation, and suggests a greater risk of harm than previously recognized.

Such risks may include psychological, social, economic, or physical harm. The guidance also distinguishes unanticipated problems from adverse events, noting that not every adverse event meets the criteria for reportable unanticipated problems.

These definitions apply within the relevant U.S. regulatory framework and should not be presented as universal rules for all countries or institutions.

Researchers should follow their institution's approved reporting procedures and consult the appropriate ethics committee when an event raises concerns about participant safety.

What If a Participant Discloses an Immediate Threat of Harm?

A disclosure suggesting imminent danger requires a different level of response from ordinary interview discomfort.

For example, a participant might communicate an immediate intention to harm themselves or another person. A chatbot should not independently improvise a crisis assessment or promise that assistance has been contacted when no such action has occurred.

The research protocol should specify how such disclosures are identified, what automated interviewing activities must stop, how human personnel are notified, and which emergency or support procedures apply.

The operational and ethical issues surrounding disclosures of imminent harm to an AI interviewer require more stringent safeguards than ordinary distress management.

Should Researchers Avoid AI Interviews Involving Sensitive Topics?

Not necessarily in every circumstance. Sensitive research can sometimes use automated tools appropriately when risks are understood and adequate safeguards are established.

However, the ethical justification becomes more demanding when interviews concern trauma, abuse, severe psychological distress, or other experiences that may require immediate contextual judgment.

Researchers should examine whether the benefits of AI-mediated interviewing justify the additional risks and whether human-led interviewing would provide more appropriate protection.

Where the system cannot reliably support the approved safety procedure, the appropriate decision may be to use a human interviewer or redesign the data collection process.

This is part of determining when AI-mediated data collection is inappropriate.

How Should Researchers Test Distress-Response Safeguards?

Safety testing should occur before participants encounter the system.

Researchers can develop hypothetical scenarios involving explicit discomfort, requests to skip questions, ambiguous emotional statements, and disclosures requiring escalation. The chatbot's responses can then be evaluated against the approved protocol.

Testing should examine whether the system stops probing when required, avoids unsupported assurances, provides appropriate participant options, and activates the correct human-contact procedure.

Researchers should also test failure conditions, such as an unavailable human contact, a disrupted internet connection, or a participant who repeatedly asks to stop.

Where the system uses generative AI, repeated testing may be necessary because a single successful response does not establish consistent performance.

These checks should form part of evaluating the safety and validity of the AI-mediated data collection process.

When an AI Interviewer Continues Probing After a Participant Expresses Distress

The chatbot acknowledges the participant's discomfort but immediately requests additional sensitive information. The expression of sympathy does not correct the inappropriate probing decision.

A more suitable response under a protocol allowing participant-controlled withdrawal might be:

The safer response does not assume that the participant requires counseling or that the system understands the experience. It respects the expressed boundary and offers choices consistent with the researcher's responsibilities.

The example illustrates an important distinction: a chatbot can produce compassionate-sounding language while still making an inappropriate interviewing decision.

Misconceptions About AI Interviewer Safety

How to Protect Participants When AI Conducts Research Interviews

Participant safety should be considered during the design of the interviewing system, not added after a distressing incident occurs.

For relatively low-risk interviews, safeguards may focus on allowing participants to skip questions, pause, withdraw, or contact the research team. Studies involving sensitive topics may require active monitoring, qualified human personnel, or a different interviewing method.

Researchers should also determine what information must be retained when a safety-related interaction occurs. Appropriate interaction records can support incident review, but unnecessary retention of sensitive disclosures may create additional privacy risks.

Informed consent should explain foreseeable risks, the role of AI, available support arrangements, and any relevant limits to confidentiality. Participants should not be led to believe that a human researcher is monitoring every message unless that monitoring actually occurs.

The central design principle is that participant welfare must take precedence over completing the interview or obtaining additional data.

Before Using an AI Interviewer With Human Participants

Frequently Asked Questions About AI Interviewers and Participant Distress

Compassionate Language Is Not a Substitute for Participant Protection

How to Cite This Guide

This guide is intended to be read, shared, and used in research, teaching, and academic work. If you draw on its ideas, explanations, or other content, please acknowledge the source by citing the guide. Doing so gives appropriate credit and helps your readers locate the original resource.