When Is AI-Mediated Data Collection Inappropriate for Research?

When Should Researchers Decide Not to Use AI for Data Collection?

Suppose you are planning interviews with university students about their experiences of discrimination, psychological distress, or academic misconduct. An AI chatbot could administer the questions, collect responses asynchronously, and reduce the time required for interviewing.

But what if a participant becomes distressed, misunderstands a question, or discloses information requiring immediate human judgment? What if the chatbot's conversational behavior changes the kind of evidence your research is intended to collect?

The availability of AI does not establish that it is appropriate for every data collection task. Researchers need to recognize circumstances in which automation may compromise the study's methodological integrity, participant welfare, or legal and ethical obligations.

AI Is Inappropriate When Its Limitations Conflict With Essential Research Requirements

Situations Where AI-Mediated Data Collection May Be Unsuitable

1. When the Research Requires Human Interpersonal Judgment

Some research interviews depend heavily on the interviewer's ability to interpret context, recognize subtle communication, and respond sensitively to participants' experiences.

For example, a phenomenological study exploring how teachers experience professional exclusion may require sustained attention to hesitation, contradiction, emotional expression, and the meanings participants develop during the conversation.

An AI system can generate follow-up questions based on the preceding text, but this does not establish that it can exercise the situated interpersonal judgment of a trained researcher.

The concern is particularly important when the interview is understood as a collaborative process through which meanings are constructed rather than a neutral mechanism for extracting information.

AI interviewing is not automatically incompatible with interpretivist research. However, replacing a human interviewer may change the interaction through which the evidence is produced. If that interaction is central to the study's epistemological assumptions, the substitution requires substantial justification.

The distinction between conversational performance and equivalence with human research interviewers should be considered before choosing autonomous interviewing.

2. When Participant Distress Requires Immediate Human Response

AI-mediated interviewing may be unsuitable when the research is reasonably likely to involve serious psychological distress and the system cannot provide safeguards proportionate to that risk.

Consider interviews about traumatic experiences, severe workplace harassment, or recent personal crises. Participants may unexpectedly become distressed, decline to continue, or require assistance beyond the scope of ordinary research interviewing.

Although a chatbot can generate sympathetic language, that capability does not establish reliable recognition of distress or appropriate human judgment.

WHO guidance on large multi-modal models identifies risks associated with inaccurate, biased, or misleading AI outputs and emphasizes the importance of oversight and context-specific evaluation in health-related applications. The guidance is not a universal prohibition on AI interviews involving sensitive topics, but its safety principles are relevant when automated responses could affect participant well-being.

Researchers should assess whether the system can stop questioning, respect participant boundaries, and activate the approved response procedure.

Where these safeguards cannot be implemented reliably, autonomous interviewing may be inappropriate even if the chatbot performs well on ordinary questions.

The specific risks of inappropriate responses to participant distress should be addressed during the study's ethical review and system evaluation.

3. When Imminent Harm Disclosures Cannot Be Managed Safely

Some studies may foreseeably elicit disclosures suggesting immediate danger to participants or other people.

For example, an AI interviewer investigating severe psychological distress might receive a statement indicating an immediate intention to cause serious harm.

If the system operates asynchronously, without qualified human monitoring or an effective escalation procedure, the research team may be unable to respond within a timeframe appropriate to the risk.

Displaying a crisis contact or sending an automated notification does not necessarily constitute effective intervention. Researchers must distinguish between an attempted alert, confirmed receipt, and an operational human response.

When foreseeable risks require timely human assessment that the study cannot provide, autonomous AI interviewing should not proceed under those conditions.

Researchers should establish the procedures for managing disclosures of imminent harm before deciding whether the interviewing design is ethically defensible.

4. When Confidentiality or Data Protection Requirements Cannot Be Met

AI-mediated data collection may involve transmitting participant responses to an external service provider. This can create additional questions about data storage, access, retention, international transfers, and secondary uses of information.

For example, a researcher may wish to collect identifiable accounts of institutional misconduct but have access only to a general-purpose chatbot service whose data processing arrangements have not been approved for the study.

In that situation, the researcher should not assume that removing participant names from the interview questions is sufficient to protect confidentiality. Participants may disclose identifying details indirectly, including workplaces, positions, personal circumstances, or recognizable events.

Data protection requirements depend on the jurisdiction, institutional arrangements, and categories of information involved.

Where the necessary legal basis, security protections, contractual safeguards, or institutional authorization cannot be established, the proposed AI service should not be used to process those participant data.

Researchers may instead use an appropriately approved environment, redesign the information collected, or adopt a method that avoids transmitting sensitive content to the external service.

5. When the AI System Changes a Standardized Measurement Instrument

Standardized questionnaires depend on carefully specified item wording, response categories, instructions, and scoring procedures.

If a chatbot independently simplifies questions, changes response anchors, or provides different explanations to different participants, the administration procedure may no longer preserve the instrument's intended measurement conditions.

For example, an established academic self-efficacy questionnaire might ask students how confident they are in completing difficult academic tasks. A chatbot that changes the item to ask whether they enjoy difficult assignments may shift the construct being assessed.

The Standards for Educational and Psychological Testing emphasize the importance of validity evidence supporting intended interpretations and uses of scores. Previously established evidence should not automatically be assumed to apply after material changes to administration.

When researchers cannot prevent or adequately evaluate such changes, generative administration may be inappropriate.

A fixed survey form or rule-based chatbot may provide a more defensible alternative when the research requires strict standardization.

6. When AI Interviewing Produces Unacceptable Measurement Bias

AI-mediated interviewing can influence responses through leading questions, uneven probing, inappropriate feedback, or systematic differences in how participants are treated.

Suppose a chatbot repeatedly asks for examples of AI-related academic misconduct but rarely requests elaboration on responsible uses. The resulting transcripts may contain more discussion of misconduct partly because the system pursued that topic more extensively.

Not every difference in probing is problematic. Semi-structured interviews legitimately permit flexibility when participants provide different information.

The concern arises when variation reflects unjustified or consequential patterns that distort the evidence.

If testing identifies persistent systematic measurement bias that cannot be adequately reduced or accounted for, the system should not be used for the intended research purpose.

7. When the System Performs Unequally Across Relevant Participant Groups

AI systems may differ in their ability to interpret languages, dialects, culturally specific expressions, disability-related communication needs, or specialized terminology.

For example, an interviewer evaluated primarily using English-language responses may not perform comparably when participants use Filipino, another Philippine language, or code-switching.

If the chatbot consistently misunderstands some participants, asks less relevant follow-up questions, or prematurely ends their responses, the study may collect systematically unequal evidence.

Researchers should assess performance in the actual linguistic and cultural context of the intended population.

When accessibility or language limitations cannot be adequately addressed, another data collection mode may be necessary to avoid excluding participants or compromising the quality of their responses.

Using AI should not require participants to abandon meaningful ways of expressing their experiences merely to accommodate the system.

8. When Participants Cannot Provide Meaningful Informed Consent

Participants should understand the nature of the research procedure, including material aspects of AI involvement, foreseeable risks, and relevant data handling arrangements.

Under the U.S. Common Rule, informed consent generally requires disclosure of information necessary for voluntary participation, subject to applicable exceptions and approved alterations. OHRP guidance emphasizes comprehension and meaningful choice rather than treating consent as a formality.

If a researcher cannot accurately explain what the chatbot does, how participant responses are processed, or whether human monitoring is available, the proposed procedure may not support meaningful informed consent.

Likewise, a system that presents itself as a human researcher without appropriate disclosure may mislead participants about the interaction.

Specific research involving justified deception or incomplete disclosure may be permissible under applicable ethical review frameworks, but methodological convenience alone does not establish such permission.

The need to disclose AI involvement to research participants should be addressed before recruitment.

9. When the AI Data Collection Process Has Not Been Adequately Validated

A chatbot's ability to conduct a convincing demonstration does not establish that it can reliably collect research evidence.

Researchers should evaluate the actual implementation against the study's requirements, including protocol adherence, question quality, response recording, bias, and relevant participant safeguards.

The NIST AI Risk Management Framework identifies validity, reliability, safety, privacy, fairness, and accountability as important characteristics of trustworthy AI systems. It also emphasizes that their relative importance depends on the context of use. The framework is voluntary and does not prescribe a single validation procedure for research interviewing.

For example, an AI interviewer might correctly administer every core question but fail to honor explicit requests to stop. Strong performance on question coverage would not compensate for that critical safeguard failure.

If the available evidence is insufficient to establish acceptable performance, researchers should delay deployment, restrict the system's functions, or select a better-supported method.

The process of validating AI-mediated data collection should precede reliance on participant data rather than serving as a retrospective justification.

10. When AI Adds Complexity Without a Defensible Research Benefit

AI-mediated data collection may also be an unsuitable choice when it introduces additional risks or complexity without improving the research procedure.

Consider a short questionnaire containing ten fixed Likert-type items. A conventional survey form can present the items, collect responses, and export data without generative conversation.

Introducing an LLM to administer the same questionnaire may create additional requirements for prompt control, response interpretation, provider agreements, logging, and validation.

If the system offers no meaningful advantage in accessibility, administration, or the study's research objectives, its use may be difficult to justify.

This does not mean AI should only be used when it demonstrably outperforms conventional methods. Researchers may legitimately investigate AI-mediated administration as a methodological innovation.

However, the added complexity should serve an identifiable research purpose rather than merely reflect the availability of the technology.

How Can Researchers Distinguish Manageable Risks From Reasons Not to Proceed?

Not every limitation requires abandoning AI-mediated data collection. Some problems can be addressed through better system design, additional testing, or human supervision.

Others may be incompatible with the study's essential requirements.

The decision should consider both the likelihood and consequences of failure. A rare but potentially severe safety failure may warrant stronger restrictions than a frequent but easily corrected formatting error.

Researchers should also distinguish risks that can be mitigated before deployment from limitations inherent to the proposed research design.

Deciding Whether AI Should Interview Students About Psychological Distress

The decision does not establish that AI is categorically prohibited in research involving academic stress. Rather, the proposed system lacks the safeguards needed for this particular study.

Suppose the researcher instead wishes to administer a brief, low-risk questionnaire about routine study habits. A controlled digital survey might be appropriate after the necessary methodological and ethical evaluation.

The same technology can therefore be appropriate for one task and inappropriate for another. The relevant unit of evaluation is the actual system operating within a particular research context.

Misconceptions About When AI Should Not Collect Research Data

How to Decide Whether AI-Mediated Data Collection Is Appropriate

Before choosing an AI interviewer or chatbot survey, identify the requirements that your data collection procedure must satisfy. These may include standardized administration, contextual probing, privacy protections, accessibility, and participant safety.

Then examine whether the proposed system can meet those requirements under the actual conditions of your study.

Researchers should document the rationale for selecting or rejecting AI-mediated data collection. Such documentation can explain why the chosen procedure was appropriate for the research question and how its limitations were addressed.

Where AI is used, preserving relevant interaction and configuration records can support subsequent evaluation without requiring unnecessary retention of sensitive information.

Choosing not to use AI is not a methodological failure. Sometimes the more defensible research decision is to retain a conventional procedure because it better serves the study's purpose.

Before Deciding to Use AI for Research Data Collection

Frequently Asked Questions About AI Data Collection Limitations

Use AI Only When the Research Can Justify Its Role

How to Cite This Guide

This guide is intended to be read, shared, and used in research, teaching, and academic work. If you draw on its ideas, explanations, or other content, please acknowledge the source by citing the guide. Doing so gives appropriate credit and helps your readers locate the original resource.