03 · What You Need to Know
When Combining AI and Human-Collected Data Is Methodologically Defensible
What Does Combining Data Actually Mean?
Combining data can refer to several different analytical decisions. Researchers may append observations into one dataset, analyze interviews together using a shared coding framework, or integrate findings from separately analyzed groups.
These approaches are not methodologically identical.
Pooling data
AI-collected and human-collected observations are treated as part of a common dataset for the intended analysis.
Integrating findings
Data from each collection mode may be analyzed separately before the findings are compared or interpreted together.
For example, a researcher may code all interview transcripts using the same thematic framework while retaining information about which interviews were AI-led. Alternatively, the researcher may compare the themes produced by each mode before developing an integrated interpretation.
Combining data does not require erasing information about how those data were collected.
Why Might AI and Human Interview Data Differ?
Research interviews are interactive processes. Even when the same core questions are used, interviewer behavior can influence what participants disclose, how much detail they provide, and which experiences receive additional attention.
A human interviewer might recognize hesitation and invite clarification. An AI interviewer might generate a follow-up question based on patterns in the participant's language. Both actions can affect the resulting transcript.
Participants may also respond differently because they know they are speaking with AI. Some may feel less personally judged, while others may question the chatbot's confidentiality or ability to understand their experiences.
Research by Pickard and Roster (2020) demonstrated that sensitive-information disclosure can differ across automated and human-facing interviewing conditions. Their findings do not establish that all AI interviews differ from human interviews, but they illustrate why interviewer presentation can matter.
Consequently, researchers should examine differences in participant responses to AI and human interviewers before assuming that collection mode is inconsequential.
Does Using the Same Interview Guide Make the Data Comparable?
Using the same core questions improves procedural consistency, but it does not establish equivalence.
For example, both interviewers might ask, "What challenges have you experienced when using AI in teaching?" Yet the human interviewer may ask for examples of institutional constraints, while the chatbot repeatedly probes technical difficulties.
The resulting transcripts would address the same broad question but might contain systematically different kinds of evidence.
Researchers should distinguish between consistency in the initial protocol and consistency in the entire interaction. The latter includes follow-up questioning, clarification, conversational feedback, and opportunities for participants to introduce unexpected topics.
This is why equivalence between AI and human research interviewers cannot be inferred solely from identical question wording.
What Is a Mode Effect?
A mode effect occurs when the method of administering a question influences the observed response independently of, or in interaction with, the characteristic being investigated.
For example, respondents might report socially sensitive behavior differently during a human-administered interview than during a self-administered digital questionnaire.
AI-mediated interviewing introduces additional possible influences through generated follow-up questions, conversational feedback, interface design, and participants' expectations about automated systems.
Not every difference between groups is a mode effect. If the AI group contains younger participants or participants from different institutions, differences may reflect group composition rather than interviewer type.
Researchers should therefore consider both collection mode and other variables that could explain observed differences.
Can Quantitative Survey Responses Be Combined?
Potentially, yes. The central question is whether the instrument operates sufficiently similarly across administration modes for the intended interpretation.
Suppose a researcher administers a 20-item Academic Self-Efficacy Scale. Half of the participants respond through a conventional human-administered procedure, while the remainder complete the questionnaire through an AI chatbot.
If the chatbot preserves the exact item wording, response categories, and scoring rules, the datasets may be structurally compatible. However, structural compatibility is not the same as measurement equivalence.
Researchers may need to examine response distributions, missing data, item functioning, reliability estimates, and measurement invariance across administration groups.
For multi-item scales, measurement invariance analysis may help determine whether the instrument's measurement structure is sufficiently comparable across groups. The appropriate level of invariance depends on whether researchers intend to compare associations, latent means, or other quantities.
Such analyses require adequate sample sizes and a measurement model appropriate to the instrument. They should not be applied mechanically to every dataset.
What About Combining Qualitative Interview Transcripts?
Qualitative interviews require a different evaluation because their value is not reducible to identical numerical measurement properties.
Researchers should examine whether both collection modes provide sufficient opportunities for participants to explain their experiences, introduce relevant topics, and clarify meanings.
Potential indicators include thematic coverage, the specificity of examples, contextual detail, and the extent to which probing shapes participant accounts.
In reflexive thematic analysis, differences between interviewing modes may be treated as part of the context in which meanings are produced rather than as errors to be statistically eliminated.
In qualitative content analysis, researchers may instead prioritize consistency in the kinds of information elicited and the coding procedures applied.
The appropriate approach depends on the study's epistemological assumptions and analytical objectives. Pooling transcripts should not imply that the interviewing context is irrelevant.
Should Collection Mode Be Recorded as a Variable?
Yes. Researchers should generally preserve a clear record of how each observation or interview was collected.
For quantitative data, a collection-mode variable might distinguish AI-administered, human-administered, and other procedures. This allows researchers to examine mode-related differences and conduct sensitivity analyses where appropriate.
For qualitative data, interview metadata can identify the interviewer type, relevant system configuration, interview protocol version, and whether human intervention occurred.
Researchers should avoid removing this information merely because they intend to analyze all responses together.
Maintaining appropriate records of AI-mediated participant interactions supports methodological transparency while allowing the research team to investigate unexpected differences.
Can Statistical Adjustment Remove Interviewer-Mode Effects?
Statistical adjustment may help account for some observed differences, but it cannot guarantee that the datasets are equivalent.
For example, including interview mode as a covariate in a regression model may help estimate associations while accounting for an observed mode-related difference under the model's assumptions.
However, adjustment does not necessarily correct measurement noninvariance, unobserved confounding, or qualitative differences in how participants interpreted questions.
If participants were not randomly assigned to interview modes, differences in participant characteristics may also complicate causal interpretation.
Researchers should distinguish between accounting for an observed statistical association and establishing that the underlying measurement process is equivalent.
What If AI and Human Data Produce Different Findings?
Differences should be investigated rather than automatically corrected or dismissed.
Suppose AI-led interviews produce more frequent criticism of institutional policies, while human-led interviews contain more detailed accounts of interpersonal support.
These differences could reflect interviewer behavior, participant expectations, sample composition, or genuine variation in experiences.
Researchers might compare probing patterns, participant characteristics, and the contexts in which particular themes emerged.
If the differences remain consequential, separate reporting may be more defensible than presenting a single pooled result.
Systematic differences may also indicate measurement bias introduced by AI-mediated interviewing, although evidence of a difference alone does not establish its cause.
How Can Researchers Decide Whether Pooling Is Justified?
The decision should begin with the intended analysis and the consequences of treating the data as comparable.
Researchers may use a pilot comparison, a randomized mode experiment, or a carefully documented assessment of existing datasets. The design should evaluate the specific features that matter for the research question.
For quantitative studies, relevant evidence may include measurement invariance, response distributions, and sensitivity analyses. For qualitative studies, researchers may examine differences in probing, thematic development, and contextual richness.
A failure to detect statistically significant differences is not proof of equivalence. Similarly, the presence of different themes does not automatically mean qualitative datasets cannot be interpreted together.
The aim is to establish whether combining the data supports a defensible interpretation, not to force every collection mode to produce identical results.
