03 · What You Need to Know
Why Participants May Respond Differently to AI and Human Interviewers
What Are Interviewer Effects?
Interviewer effects occur when characteristics or behaviors associated with the interviewer influence participants' responses.
In conventional interviewing, these effects may arise from an interviewer's perceived authority, identity, manner of questioning, reactions, or relationship with the participant.
For example, participants may avoid admitting socially undesirable behavior when they believe a human interviewer will judge them. They may also elaborate more readily when an interviewer demonstrates understanding and interest.
Replacing a human interviewer with AI does not necessarily eliminate these effects. Instead, it changes the social and technological context in which responses are produced.
Participants may perceive an AI interviewer as less judgmental, more impersonal, less trustworthy, or less capable of understanding their experiences. These perceptions can shape their answers even when the interview questions remain identical.
Do Participants Disclose More Sensitive Information to Automated Interviewers?
Some empirical studies suggest that automated interviewing can encourage disclosure of information participants might hesitate to share with a human interviewer.
Pickard, Roster, and Chen (2016) investigated preferences for disclosing sensitive information to human interviewers and embodied conversational agents. In their scenario-based study involving 203 undergraduate business students, participants generally preferred an automated avatar for highly sensitive topics involving potentially negative self-disclosure. Human interviewers were preferred for less sensitive topics.
Participants' explanations suggested that perceived judgment was an important factor. Some regarded an automated interviewer as less capable of evaluating them personally.
However, this study primarily examined preferences in hypothetical disclosure scenarios. Preference for an interviewer should not be interpreted as direct evidence that participants would necessarily disclose more accurate information during actual interviews.
A subsequent experiment by Pickard and Roster (2020) examined actual disclosure among 158 university students randomly assigned to three interviewing conditions: audio computer-assisted self-interviewing, an embodied conversational agent, or a human interviewer.
The researchers found greater disclosure in the audio-only automated condition than in the two conditions featuring a human or human-like face. They did not find a statistically significant difference between the embodied conversational agent and human interviewer conditions.
This distinction is important. The results suggest that removing a visible interviewer may influence disclosure, but merely replacing a human face with a virtual one does not necessarily produce the same effect. The experiment demonstrates why interviewer modality must be examined more carefully than a simple AI-versus-human comparison.
What Is Social Desirability Bias in AI-Mediated Interviews?
Social desirability bias occurs when respondents provide answers that present themselves favorably rather than fully reflecting their actual behaviors, experiences, or beliefs.
For example, a teacher might underreport using AI to prepare assessments without checking their accuracy because they believe admitting this behavior would be professionally embarrassing.
A human interviewer may increase perceived evaluation pressure in some circumstances. An automated system might reduce that pressure if participants believe it cannot judge them.
However, AI systems can also create social expectations. A chatbot that uses a human-like avatar, personalized praise, or highly responsive language may encourage participants to treat the interaction as socially evaluative.
Research published in Decision Support Systems in 2018 examined conversational agent embodiment and conversational relevance in relation to socially desirable responding. The study found that participants disclosed less sensitive behavior to human interviewers than to conversational agents. It also found that agents appearing to understand participants could reduce disclosure relative to less conversationally responsive agents.
The finding complicates the assumption that making a chatbot more human-like necessarily improves research data collection.
Importantly, lower social desirability does not automatically establish greater factual accuracy. Researchers need an appropriate basis for evaluating whether reported information corresponds to participants' actual experiences or behaviors.
Does the Chatbot's Appearance Affect What Participants Say?
Yes, interface characteristics may influence participant perceptions and responses.
An AI interviewer can appear as a text-only chat interface, a synthetic voice, an animated avatar, or a more elaborate virtual character. These designs may produce different expectations about privacy, social presence, and conversational competence.
For example, a participant might regard a text-only chatbot as a relatively impersonal reporting mechanism. An avatar with a face and expressive responses may instead create a stronger sense of interpersonal interaction.
The experiment by Pickard and Roster (2020) is particularly relevant because it found that disclosure differed between faceless automated interviewing and interviewer conditions involving visible faces.
Researchers should therefore avoid treating all AI interviewing interfaces as interchangeable. The interface itself can be part of the data collection condition.
Do Participants Provide Longer or More Detailed Answers to Human Interviewers?
Not necessarily. Response length and detail can vary according to interview format, topic, conversational design, and participant characteristics.
Human interviewers may encourage elaboration through contextual probing and interpersonal engagement. AI systems may also generate follow-up questions that prompt participants to provide additional examples or explanations.
However, longer responses are not automatically more accurate, relevant, or analytically useful. An interview containing many words may still provide limited insight into the research question.
Researchers should distinguish response quantity from substantive quality. Useful indicators might include whether participants provide concrete examples, explain relevant circumstances, acknowledge uncertainty, and introduce experiences not anticipated by the interview protocol.
Evidence concerning AI-generated follow-up questions suggests that probing behavior itself requires evaluation because it can shape the detail and direction of participants' responses.
Can Participants Feel More Comfortable With AI but Trust It Less?
Yes. Comfort and trust are related but distinct constructs.
A participant might feel less embarrassed discussing sensitive experiences with an automated interviewer while simultaneously worrying about how the system stores or processes personal information.
Conversely, a participant may trust a human researcher's professional judgment but hesitate to disclose behavior that could invite criticism.
These apparently contradictory responses are plausible because participants evaluate several aspects of the interaction: perceived judgment, confidentiality, competence, interpersonal understanding, and control over their information.
Researchers should not assume that willingness to use an AI interviewer establishes confidence in its privacy protections or satisfaction with the interviewing experience.
Does the Research Topic Change How Participants Respond?
Topic sensitivity is an important potential moderator of interviewer effects.
Questions about routine academic practices may generate relatively similar responses across interviewing modes. Questions involving misconduct, personal distress, institutional criticism, or socially stigmatized experiences may produce different disclosure patterns.
Pickard, Roster, and Chen (2016) found that preferences for human versus automated interviewers differed according to the perceived sensitivity of the topic. This suggests that researchers should not generalize findings from ordinary factual questions to emotionally or socially sensitive interviews.
Participant characteristics may also matter. Familiarity with AI, digital literacy, language proficiency, accessibility needs, and previous experiences with automated systems could influence how people interpret and respond to chatbot questions.
These possibilities should be evaluated rather than assumed to operate identically across demographic or cultural groups.
Can Participants Change Their Responses Because They Know the Interviewer Is AI?
Yes. Knowledge about interviewer identity may influence expectations and behavior.
Participants who know they are interacting with AI may provide different responses from those who believe they are communicating with a human researcher. They may also adjust their language because they anticipate that the system has difficulty understanding sarcasm, ambiguity, or culturally specific expressions.
Researchers should nevertheless avoid concealing AI involvement merely to reduce possible interviewer effects. The need for transparent disclosure of AI interviewing remains an important ethical consideration.
If interviewer identity is itself an experimental variable, any proposed concealment or deception requires specific ethical justification and applicable institutional review.
How Can Researchers Determine Whether Interviewer Type Changes Responses?
A comparative design can help isolate interviewer-mode effects.
Researchers might randomly assign participants to AI-led or human-led interviews using equivalent core questions. They could then examine differences in response content, disclosure, completion, perceived comfort, and the quality of follow-up interactions.
For quantitative outcomes, the analysis should reflect the outcome's measurement level, distribution, and study design. For qualitative outcomes, researchers may compare thematic coverage, contextual detail, contradictions, and the emergence of unanticipated experiences.
Random assignment can strengthen causal interpretation, but it does not remove every source of variation. The comparison may still depend on the particular human interviewers, AI model, interface, protocol, and participant population.
Researchers should also avoid interpreting the absence of a statistically significant difference as proof that both interview modes are equivalent.
The broader question of AI-human interviewer equivalence requires criteria that extend beyond any single measure of response behavior.
04 · A Practical Example
Would Teachers Disclose More AI-Related Misconduct to a Chatbot?
Hypothetical Example
Comparing Disclosure Across Interview Modes
A researcher investigates whether university faculty disclose potentially questionable uses of generative AI differently depending on interviewer type.
Sixty eligible faculty members are randomly assigned to two groups. Thirty participate in interviews conducted by a trained human researcher, while thirty complete interviews through a text-based AI chatbot.
Both groups receive the same core questions about AI use, verification practices, and experiences with institutional guidelines.
Step 1: Define the outcome
The researcher identifies a disclosure indicator: whether participants acknowledge using AI-generated instructional materials without independently verifying their accuracy.
Step 2: Standardize the questions
Both groups receive the question, "Have you ever used AI-generated instructional materials without checking their accuracy before classroom use?"
Step 3: Collect responses
The human interviewer follows a standardized protocol. The chatbot presents the same questions without generating evaluative comments or praise.
Step 4: Compare disclosure
Suppose 9 of 30 participants in the human-led condition and 15 of 30 participants in the AI-led condition acknowledge the behavior.
Step 5: Interpret the result
The observed disclosure rates are 30% for human interviews and 50% for AI interviews, a difference of 20 percentage points.
In this hypothetical study, participants interviewed by AI disclose the specified behavior more frequently. However, the difference does not establish that the AI responses are more truthful or that AI interviewing is superior.
The sample is small, and the researcher would need to examine statistical uncertainty, the interview procedures, and alternative explanations. More importantly, without an independent criterion for actual behavior, the study measures reported disclosure rather than verified honesty.
The finding could motivate further investigation into whether perceived judgment, confidentiality expectations, or conversational behavior contributed to the difference.
