Can AI Make Sensitive Inferences That Researchers Never Intended to Collect?

Can AI Generate Personal Information That Was Never Collected From Participants?

You are analyzing interviews with university instructors about their experiences with generative AI in teaching. The study collects information about instructional decisions, institutional expectations, and professional practices.

You ask an AI system to summarize the transcripts and identify patterns in participants' responses.

Alongside the requested categories, the model begins describing certain instructors as emotionally insecure, technologically anxious, or professionally resistant. It may even suggest that particular writing patterns reveal personal characteristics that participants never discussed.

These interpretations were not part of the interview guide. Participants were not asked to disclose such characteristics, and the research protocol did not identify them as variables or phenomena of interest.

Nevertheless, the AI system has produced additional information about participants, or at least claims that appear to describe them.

Can generative AI create ethically significant inferences from existing research data even when researchers never intended to collect those characteristics?

Yes, AI Can Generate Sensitive Inferences Beyond the Original Research Purpose

How AI Can Generate Sensitive Information From Existing Research Data

What Is a Sensitive Inference?

A sensitive inference is a conclusion or prediction about a potentially sensitive characteristic derived from other available information rather than directly obtained through an explicit disclosure or measurement of that characteristic.

For example, a participant may describe experiencing difficulties with institutional AI policies. An AI model might infer that the participant has a particular psychological condition or possesses a particular personal belief.

The participant did not necessarily disclose that characteristic, and the interview may provide no defensible basis for determining it.

Inferences can concern a wide range of characteristics, including health-related information, personal beliefs, financial circumstances, professional competence, or other attributes that may carry privacy or social consequences.

Whether an inferred characteristic is legally classified as sensitive personal information depends on the applicable jurisdiction and statutory definitions.

Researchers should therefore distinguish the ordinary meaning of a sensitive inference from the specific legal categories of sensitive or special-category personal data.

How Can AI Infer Information That Was Never Explicitly Provided?

Generative AI can interpret linguistic patterns, contextual clues, relationships among statements, and information supplied in prompts.

For example, a participant might discuss financial pressures, teaching responsibilities, or uncertainty about institutional policies. The model may combine these details into a broader interpretation of the participant's circumstances.

However, the ability to generate a plausible interpretation does not establish that the inference is accurate.

AI systems may rely on stereotypical associations, incomplete contextual information, or patterns that are not sufficiently informative about the individual.

Research on privacy inference has demonstrated that seemingly ordinary information can reveal or support predictions about characteristics not directly disclosed. Kosinski, Stillwell, and Graepel (2013), for example, examined how digital records could predict certain personal attributes.

That study concerned digital behavioral records and predictive models rather than generative AI interview analysis specifically. Its relevance lies in demonstrating the broader possibility of inferring personal attributes from indirect information.

Distinguish Observed Data, Analytical Interpretation, and Sensitive Inference

Not every interpretation generated from research data is an inappropriate inference.

Researchers routinely interpret participant statements, identify patterns, and develop analytical explanations. The concern arises when an interpretation extends into personal characteristics that the evidence or research purpose does not justify.

The distinction is not determined solely by the wording. A personal characteristic may be a legitimate object of inquiry in one study and an unjustified inference in another.

The researcher must establish the evidential basis and methodological purpose of the interpretation.

What Types of Sensitive Inferences Might AI Generate?

AI systems may generate several kinds of personal interpretations from research material, sometimes without being explicitly asked to do so.

These are possible inference risks, not evidence that a particular generative AI system can reliably determine the characteristics listed.

Researchers should be especially cautious when a model presents uncertain personal interpretations in confident language.

Why an AI Inference Is Not Equivalent to a Measured Variable

Researchers may be tempted to treat AI-generated characteristics as additional variables available for analysis.

For example, a model might assign each interview participant a level of anxiety, professional confidence, or technological competence based on their statements.

However, those classifications are not automatically equivalent to scores obtained through an appropriately developed and evaluated research instrument.

Measurement requires a defensible operational definition, suitable procedures, and evidence supporting the intended interpretation and use of results.

An AI-generated label does not establish construct validity merely because it appears plausible or consistent across several model responses.

Researchers should not convert speculative interpretations into participant characteristics without addressing the necessary measurement and ethical requirements.

This is particularly important when the inferred variable was not included in the original research design.

Can Sensitive Inferences Be Created Accidentally?

Yes. Researchers may ask AI to perform a legitimate analytical task and receive additional interpretations that exceed the requested scope.

For example, a prompt requesting patterns in instructors' approaches to AI-assisted teaching may produce categories involving personality, emotional vulnerability, or presumed professional attitudes.

The researcher may not have requested these classifications, but they may still appear in the output.

The ethical significance depends partly on what happens next. An incidental unsupported statement that is recognized and excluded differs from an inference that is systematically generated, retained, linked to participants, and incorporated into findings.

Researchers should not treat accidental generation as permission to expand the study's analytical purpose.

They should also consider whether recurring unwanted inferences indicate that the prompt or workflow needs to be revised.

Does an Inference Become Personal Data?

Potentially. Under frameworks such as the European Union's General Data Protection Regulation, personal data include information relating to an identified or identifiable natural person.

An inferred characteristic linked to an identifiable participant may therefore fall within the scope of personal data processing even when the characteristic was generated by an algorithm rather than supplied directly by the individual.

Additional restrictions may apply when the processing concerns legally defined special categories of personal data.

However, the precise classification depends on the information, processing context, and applicable law. Not every speculative inference automatically falls within every special-category definition.

In the Philippines, the Data Privacy Act of 2012 defines personal information and sensitive personal information and establishes relevant processing requirements.

Researchers should consult the applicable legal and institutional framework rather than assume that AI-generated information is outside privacy regulation because it was not directly collected from participants.

Why the Original Research Purpose Matters

Research protocols generally identify the information to be collected and the purposes for which it will be used.

Suppose a study investigates instructors' experiences of making decisions about generative AI. The researcher collects interviews about teaching practices and institutional expectations.

Using those interviews to infer participants' health conditions or personal beliefs may represent a different analytical purpose.

Even if the inference could be generated from existing information, that does not establish that the additional processing is necessary or authorized.

Researchers should consider purpose limitation, data minimization, consent arrangements, and the approved protocol.

The concern is closely related to whether AI-assisted analysis violates data minimization principles.

Can Sensitive Inferences Introduce Discrimination or Stigmatization?

Yes. Unsupported personal inferences may lead to unfair or stigmatizing descriptions of individuals and groups.

For example, an AI system might characterize instructors who express uncertainty about generative AI as psychologically insecure or professionally incapable.

If such classifications are reported as findings, they may misrepresent participants and encourage unsupported judgments.

The risks may be greater when participants are identifiable or when the findings influence institutional decisions.

However, researchers should distinguish the possibility of harm from evidence that a particular discriminatory consequence has occurred.

Potential discrimination and stigmatization risks should be examined in relation to the actual inference and its intended use.

What If AI Infers a Characteristic Accurately?

Even an apparently accurate inference may raise ethical concerns.

Accuracy and authorization are different questions. An inference may correspond to a person's circumstances while still exceeding the purpose for which the information was provided.

For example, a participant may disclose professional experiences in a confidential interview without agreeing to have unrelated personal characteristics inferred from those statements.

Researchers should therefore consider whether generating and using the inference is necessary and ethically justified, not merely whether it might be correct.

Moreover, apparent accuracy should not be accepted without appropriate evidence. A model's confidence or explanatory fluency does not establish that its inference has been validated.

Can Researchers Use AI to Infer Characteristics for Exploratory Analysis?

Exploratory research may legitimately investigate relationships or develop new analytical questions. However, the exploratory label does not automatically authorize every form of personal inference.

For example, researchers may notice that participants describe different forms of professional uncertainty. Exploring those accounts could be appropriate within a qualitative inquiry.

However, asking AI to infer undisclosed psychological conditions from the same interviews would involve a different kind of claim and potentially different ethical requirements.

Researchers should distinguish exploration of participant-reported experiences from classification of individuals according to unmeasured characteristics.

When the proposed analysis substantially changes the research purpose or risks, institutional review may be necessary.

How Can Researchers Limit Unnecessary AI Inferences?

One approach is to define the AI task narrowly and explicitly exclude personal profiling unrelated to the research questions.

For example, instead of asking AI to "analyze everything you can infer about these participants," the researcher might request a summary of statements concerning specific instructional decisions.

The prompt can instruct the model not to infer health conditions, demographic characteristics, personality traits, or other personal attributes not explicitly relevant to the approved analysis.

However, prompt instructions are not a guarantee. The system may still produce unintended interpretations.

Researchers should review outputs, avoid retaining unnecessary sensitive classifications, and follow applicable procedures for handling information already generated.

Technical safeguards and restrictions on the data submitted may be more dependable than relying exclusively on the model's compliance with a prompt.

How Should You Prompt AI to Avoid Sensitive Participant Inferences?

A useful prompt limits the analysis to the stated research purpose and requires clear separation between participant statements and researcher interpretations.

This prompt may help constrain the output, but it does not establish that the AI system is suitable for processing the underlying research data.

Researchers must first verify that the data and processing arrangement are appropriately authorized.

What Should Researchers Do When AI Produces an Unwanted Sensitive Inference?

The researcher should first determine whether the output represents a legitimate interpretation within the approved study or an unnecessary personal inference.

If the inference is unsupported or unrelated to the research purpose, it should not be incorporated into findings or treated as verified participant information.

Researchers should consider whether the output needs to be excluded, corrected, or handled under applicable data management procedures.

Where sensitive information has been generated and stored in identifiable form, the research team may need to evaluate retention, access, and deletion requirements.

If the activity reveals that the AI workflow is systematically generating information outside the approved purpose, the researcher should reassess the procedure and consult the appropriate institutional authority where necessary.

The objective is not merely to remove an uncomfortable label from the final manuscript. It is to ensure that the analytical process itself remains appropriately limited.

Can AI Inferences Change the Ethical Risk of an Approved Study?

Yes. Introducing a procedure that generates additional sensitive characteristics may change the scope and consequences of data processing.

For example, an approved study may concern instructors' experiences with AI-assisted teaching. A subsequent decision to infer participants' psychological characteristics from interview transcripts may introduce a new analytical purpose and potential risks.

Whether the change requires amendment, notification, or additional review depends on institutional requirements and the significance of the proposed activity.

Researchers should examine how AI can change research ethical risks and avoid assuming that all analyses derived from previously collected data are automatically covered by the original approval.

Should Researchers Tell Participants About AI-Generated Inferences?

Participant information should accurately describe material aspects of the research procedures, including relevant analytical purposes and processing arrangements.

Whether particular inferences require specific disclosure or consent depends on the research design, applicable law, and institutional requirements.

However, researchers should not use the possibility of disclosure as a substitute for determining whether the inference should be generated at all.

For example, informing participants that AI may infer unrelated sensitive characteristics does not automatically justify collecting those characteristics.

The researcher should first establish necessity, proportionality, methodological justification, and authorization.

The relationship between formal permission and participant expectations concerning AI processing remains relevant when the proposed analysis extends beyond information participants knowingly provided.

When AI Infers Psychological Characteristics From Teaching Interviews

The issue is not that psychological characteristics can never be investigated in research. Rather, this particular study did not collect evidence sufficient to establish them or identify them as part of its analytical purpose.

The example illustrates how AI can transform an ordinary statement about professional practice into an unsupported personal inference.

Common Misconceptions About AI-Generated Sensitive Inferences

Do Not Treat AI-Generated Personal Inferences as Additional Research Data by Default

When AI assists with research analysis, examine whether its outputs describe information participants provided or introduce characteristics that were never part of the study.

Do not assume that every inference is useful simply because it can be generated from the available material.

Document consequential decisions about AI-generated inferences, including the evidence used to evaluate them and the reasons for exclusion or retention.

The researcher's responsibility is not only to determine whether an inference is plausible, but whether the study has a legitimate and defensible reason to generate it.

Before Using AI-Generated Inferences About Research Participants

Frequently Asked Questions About AI Sensitive Inferences in Research

AI Can Infer More Than Participants Disclosed, but That Does Not Mean Researchers Should Use It

How to Cite This Guide

This guide is intended to be read, shared, and used in research, teaching, and academic work. If you draw on its ideas, explanations, or other content, please acknowledge the source by citing the guide. Doing so gives appropriate credit and helps your readers locate the original resource.