What Should Researchers Log From an AI-Mediated Participant Interaction?

What Records Should You Keep When AI Interacts With Research Participants?

Suppose an AI chatbot conducts interviews with 50 university teachers. You save their responses and begin analyzing the transcripts. Later, you notice that several participants discussed academic misconduct in considerable detail, while others barely mentioned it.

Did the participants naturally differ in their experiences, or did the chatbot ask different follow-up questions? Was the same AI model used throughout the study? Did a system update change how questions were generated?

Without appropriate interaction records, these questions may be impossible to answer. Yet recording everything indiscriminately can create unnecessary privacy risks, particularly when participants disclose sensitive information.

Record What Is Necessary to Reconstruct and Evaluate the Interaction

Which AI-Participant Interaction Records Matter for Research?

Why Are Interaction Logs Different From Research Transcripts?

A research transcript records the conversation or participant responses that form part of the study's evidence. An interaction log documents relevant events and system behavior associated with producing that evidence.

For example, a transcript might show that a participant described difficulty using generative AI in teaching. An interaction log might additionally identify the interview protocol version, the AI model used, whether a follow-up question was generated dynamically, and whether the interaction experienced a technical interruption.

These records may overlap. A timestamped transcript containing all questions and responses can provide part of the audit trail. Researchers do not necessarily need to duplicate the same sensitive content across several files.

The distinction is functional: the transcript supports analysis of participant accounts, while the audit record supports examination of the data collection procedure.

What Information Should Researchers Consider Logging?

The following categories provide a practical starting point. Not every field is necessary for every study, and some should be omitted when their privacy costs exceed their methodological value.

Researchers should determine which fields are necessary before data collection begins. Logging requirements should be documented in the research protocol and aligned with the approved data management arrangements.

Should You Record Every Question the AI Actually Asked?

For generative interviewing, preserving the questions actually presented is particularly important.

A planned interview guide establishes what the researcher intended the chatbot to ask. It does not establish what the participant actually encountered.

For example, the approved question might be, "What challenges have you experienced when using AI for teaching?" The chatbot could instead ask, "How has unreliable AI affected your teaching?" The second formulation introduces an assumption that may influence the participant's response.

If only the participant's answer is retained, the researcher may not recognize the influence of the preceding question.

Records of dynamically generated follow-up questions therefore support evaluation of relevance, neutrality, and protocol compliance.

For fixed-question surveys, a verified questionnaire version and an accurate record of item sequence may sometimes provide sufficient documentation without storing redundant copies of every displayed item.

Should Researchers Save the System Prompt?

When prompts or system instructions materially determine the interviewing procedure, researchers should preserve an appropriately controlled record of the instructions actually used.

These instructions may specify the chatbot's role, approved topics, permissible probes, prohibited behaviors, stopping rules, and escalation procedures.

A record of the model name alone is insufficient to reconstruct the interviewing configuration when different prompts could produce substantially different behavior.

However, prompts may contain confidential institutional information, proprietary instructions, or security-sensitive implementation details. They should not automatically be included in public datasets.

Researchers can preserve a restricted configuration record and use version identifiers or cryptographic hashes to associate individual sessions with the applicable instructions.

A hash can help identify whether a recorded configuration has changed, but it cannot reconstruct the original instructions if the underlying file has been lost.

Why Should the AI Model and Version Be Documented?

Generative AI systems can behave differently across model families, versions, configurations, and provider updates.

A researcher who records only "ChatGPT was used" may leave insufficient information for readers to understand the actual interviewing procedure.

Where available, relevant documentation may include the provider, model identifier, model version or snapshot, deployment environment, and generation settings that materially affect behavior.

Researchers should also record when a model or configuration changes during data collection.

Even detailed documentation may not make a generative interaction exactly reproducible. Providers may update underlying systems, and nondeterministic generation can produce different outputs under apparently similar conditions.

The purpose of documentation is therefore to support scrutiny and reasonable reproducibility, not to promise that every conversational exchange can be regenerated identically.

Should You Log AI Decisions That Participants Never See?

Some internal decisions may be methodologically consequential even when they are not displayed to participants.

For example, the system might classify an answer as complete, decide to skip a follow-up question, or select the next topic based on an automated interpretation.

Where these decisions affect the information collected, researchers should consider recording the decision outcome and the relevant rule or configuration.

However, this does not mean researchers must retain unrestricted model reasoning traces or hidden internal processes. Such information may be unavailable, unreliable as an explanation, proprietary, or unnecessary for the research purpose.

Observable decision records, such as "follow-up omitted because maximum probing limit was reached," may provide more defensible documentation than attempting to treat generated explanations as direct access to a model's internal reasoning.

What About Errors, Interruptions, and Protocol Deviations?

Technical and procedural failures can affect the completeness and interpretation of research data.

For example, a chatbot might skip a required question, lose part of a participant's response, repeat an item, or terminate unexpectedly.

Researchers should record consequential deviations and determine whether affected data remain suitable for analysis.

Useful information may include the nature of the event, the affected session, the relevant point in the interaction, and any corrective action.

Not every minor technical event requires extensive documentation. The appropriate level of detail depends on whether the event could affect participant welfare, data integrity, or the intended interpretation.

How Should Safety-Related Events Be Logged?

Safety-related records require particular care because they may contain highly sensitive information.

Researchers may need to document explicit withdrawal requests, inappropriate probing after distress, activated escalation procedures, or failures to contact designated personnel.

Where a participant makes a disclosure suggesting imminent harm during an AI interview, the approved safety procedure should determine what information is necessary for response and subsequent review.

Records should distinguish between an attempted action and a confirmed outcome. For example, "notification sent" is not equivalent to "notification received and acted upon."

Safety logs should be accessible only to appropriately authorized personnel and retained according to applicable requirements.

Should Researchers Collect IP Addresses and Other Technical Identifiers?

Not automatically. Technical identifiers may be useful for certain security, eligibility, or fraud-prevention purposes, but they can also increase identifiability.

IP addresses, device identifiers, account information, and precise location data may constitute personal data depending on the circumstances and applicable law.

Researchers should determine whether these identifiers are genuinely necessary, whether less intrusive alternatives are available, and how the information will be protected.

For studies intended to be anonymous, researchers should verify whether the platform or external provider collects identifying metadata even when the research questionnaire does not ask for names.

Calling a dataset anonymous does not make it anonymous if records can reasonably be linked to identifiable individuals.

How Long Should Interaction Logs Be Retained?

There is no universal retention period applicable to every research interaction log.

Retention may depend on institutional research policies, funder requirements, ethics approval, contractual obligations, applicable law, and the nature of the data.

Researchers should distinguish between records necessary for scientific accountability and temporary operational logs that may no longer serve a legitimate purpose.

For example, a study may need to retain a documented interview protocol and de-identified transcripts for an approved period while deleting unnecessary raw diagnostic records sooner.

The NIST Privacy Framework emphasizes managing privacy risks across data processing activities. Its principles support considering data minimization, access, retention, and disposal as part of system design.

In the Philippines, the Data Privacy Act of 2012 establishes obligations relevant to personal information processing. Researchers should consult applicable institutional policies and data protection guidance rather than assuming that a single retention rule applies to all research records.

Can Logs Help Detect AI Interviewing Bias?

Yes. Interaction records can reveal patterns that would be difficult to identify from participant responses alone.

For example, researchers might compare how frequently the chatbot asks follow-up questions after positive and negative statements, whether certain participants receive more clarification requests, or whether specific topics are consistently emphasized.

Such patterns may indicate differences in interview administration requiring investigation.

However, a difference in probing frequency does not automatically establish bias. Some responses genuinely require more clarification than others.

Researchers should evaluate whether observed differences are justified by the content of responses or suggest systematic measurement bias.

Interaction logs provide evidence for this evaluation, but they do not independently establish the validity of the collected data.

How Should Interaction Logs Be Organized?

A practical arrangement separates participant-facing research content, system configuration records, and operational event records.

These components can be linked through pseudonymous session identifiers and controlled version references.

Researchers should establish access permissions appropriate to each category. For example, analysts may need de-identified transcripts but not identifiable contact information or sensitive incident reports.

The data management plan should specify what is collected, why it is necessary, who can access it, how it is protected, and when it will be deleted or archived.

Such documentation contributes to validation of the AI-mediated data collection process by making it possible to examine the system's actual behavior and investigate consequential deviations.

Designing an Audit Trail for an AI Interview Study

An illustrative event record might contain the following fields:

Suppose the researcher later discovers that participants interviewed under configuration 03 received more follow-up questions than those interviewed under configuration 02.

Because the configuration and question sequence were documented, the researcher can investigate whether the difference resulted from an intentional change, participant responses, or unexpected model behavior.

Without those records, the difference might remain invisible or be incorrectly attributed to participants.

Misconceptions About AI Research Interaction Logs

How Much Logging Does Your AI Research System Actually Need?

The appropriate logging strategy depends on the degree of AI autonomy and the consequences of errors.

A fixed-question chatbot may need little more than accurate responses, questionnaire version information, and relevant administration events. A generative interviewer may require a more detailed record of the questions it generated and the configuration governing those decisions.

Researchers should establish logging requirements before deployment and review them when the system or research protocol changes.

Records should be sufficient to support methodological scrutiny, but they should not expose participants to avoidable privacy risks merely for the sake of technical completeness.

Before Finalizing Your AI Interaction Logging Plan

Frequently Asked Questions About AI Research Interaction Logs

Keep Enough Evidence to Explain What the AI Actually Did

How to Cite This Guide

This guide is intended to be read, shared, and used in research, teaching, and academic work. If you draw on its ideas, explanations, or other content, please acknowledge the source by citing the guide. Doing so gives appropriate credit and helps your readers locate the original resource.