Can AI Introduce New Risks of Discrimination or Stigmatization in Research?

Can AI Turn Ordinary Research Findings Into Unfair Judgments About Participants?

You are analyzing interviews with university instructors about their decisions concerning generative AI. Some participants describe frequent AI use, while others explain that they rarely use it or have chosen not to incorporate it into teaching.

You ask an AI system to identify differences among participants. It organizes their responses into categories such as "innovative adopters," "hesitant users," and "technology-resistant instructors."

The categories appear useful because they summarize different approaches to AI. However, one label carries an interpretation that the interviews may not support. An instructor who avoids AI for pedagogical reasons has been characterized as resistant to technology.

If these labels appear in the research findings, they may portray some participants as less competent, less progressive, or less professionally capable without adequate evidence.

Can generative AI introduce discrimination or stigmatization into research even when the researcher never intended to make unfair judgments?

Yes, AI Can Introduce Unfair Classifications and Stigmatizing Interpretations

How AI Can Create Discrimination and Stigmatization Risks in Research

What Is the Difference Between Discrimination and Stigmatization?

Discrimination and stigmatization are related concerns, but they are not identical.

Discrimination generally involves unfair or unjustified differential treatment, exclusion, or disadvantage associated with particular characteristics or group membership.

Stigmatization involves attaching discrediting meanings or negative social judgments to individuals or groups. It may contribute to discrimination, but a stigmatizing representation does not necessarily demonstrate that discriminatory treatment has already occurred.

For example, describing instructors who do not use AI as professionally outdated may contribute to stigmatization. If that classification is subsequently used to deny professional development opportunities or evaluate employment performance unfairly, discriminatory consequences may arise.

Researchers should distinguish the harmful potential of a label from evidence that a particular discriminatory outcome has occurred.

How Can Generative AI Produce Biased Research Categories?

Generative AI systems generate outputs from patterns learned during training and information supplied during interaction. These patterns may reflect stereotypes, unequal representation, or common assumptions about particular groups.

When asked to classify research participants, a model may reproduce familiar categories without establishing whether those categories are appropriate to the study.

For example, it may interpret limited AI use as resistance, reluctance, or low technological competence. Yet participants may have legitimate reasons for nonuse, including instructional objectives, accessibility concerns, institutional restrictions, or uncertainty about evidence of effectiveness.

The problem is not merely that the classification sounds negative. It is that the label may introduce an explanation not established by the participant's account.

Barocas and Selbst (2016) demonstrate how data-driven decision-making can reproduce discriminatory effects through choices concerning data, variables, and analytical procedures. Their work addresses big data systems more broadly, but its analysis is relevant to examining how AI-generated categories may create unfair distinctions.

Where Can Discrimination and Stigmatization Enter the Research Process?

Potential problems can emerge at different stages of AI-assisted research.

The risks differ in nature and severity. A preliminary coding suggestion that is critically reviewed may have different consequences from an automated classification used to determine eligibility for a research intervention.

AI Can Confuse a Research Description With a Personal Judgment

Researchers often need to describe differences among participants. However, descriptive categories should not automatically become evaluations of character or competence.

Consider the following distinction.

The alternatives are not automatically superior in every study. If participants explicitly describe resistance or fear, those experiences may legitimately become part of the analysis.

The researcher must establish whether a label represents participants' own accounts, a justified analytical interpretation, or an unsupported inference introduced by AI.

Can AI Misinterpret Language and Cultural Context?

Yes. AI-generated interpretations may be affected by linguistic variation, cultural expressions, and differences in how participants communicate.

For example, a participant may use indirect language to express disagreement or employ expressions whose meaning depends on a particular cultural or institutional context.

An AI system may interpret the statement literally or assign an emotional category that does not reflect the participant's intended meaning.

These errors may become particularly consequential if they occur more frequently for particular linguistic groups.

Researchers should examine whether the model adequately handles the languages, dialects, and contexts represented in the data.

They should not assume that grammatical fluency in a language establishes cultural competence or reliable interpretation.

Blodgett and colleagues (2020) discuss how bias in language technologies must be understood in relation to social power, language variation, and the harms associated with system behavior.

What Is the Difference Between Bias and Demonstrated Discrimination?

AI bias may refer to systematic patterns of error, unequal representation, or assumptions that favor certain interpretations.

Discrimination concerns unjustified disadvantage or differential treatment, which may arise from biased systems but is not established by every instance of model error.

For example, if an AI model incorrectly labels one instructor's statement as resistant, the researcher has identified a possible interpretive error.

If the model systematically applies that label to a particular participant group and the classification affects institutional decisions, the concern becomes more consequential.

Researchers should therefore avoid claiming that discrimination has occurred solely because an AI output appears inaccurate or unfavorable.

At the same time, they should not dismiss systematic errors simply because no immediate harm has yet been documented.

Can AI Create New Sensitive Inferences?

Generative AI may infer personal characteristics that were not part of the original data collection.

For example, a model might interpret an instructor's comments about workload as evidence of psychological distress or infer professional competence from writing style.

Such interpretations may be unsupported and could introduce new risks if they are used to characterize participants.

Researchers should examine whether the inferred characteristic is necessary for the research purpose and whether the available evidence supports it.

They should also consider whether the inference involves sensitive information or creates risks of unfair treatment.

The broader concern is examined in AI-generated sensitive inferences from research data.

Can AI Discrimination Occur Without Using Protected Demographic Variables?

Potentially. Excluding explicit demographic variables does not necessarily eliminate the possibility of unfair group-related effects.

Other information may function as a proxy for characteristics associated with particular populations.

For example, institutional affiliation, geographic location, language use, or educational background may correlate with characteristics that were not directly included in the analysis.

An AI system could generate classifications that systematically disadvantage particular groups even when those groups are not explicitly named in the prompt.

However, the existence of a correlation does not automatically establish discrimination. Researchers should examine the actual classification process, its justification, and its consequences.

Barocas and Selbst (2016) provide a foundational discussion of how apparently neutral data-driven practices can produce disparate impacts.

How Should Researchers Evaluate AI-Generated Categories?

Researchers should examine whether categories are grounded in the data, conceptually justified, and appropriate to the analytical approach.

In qualitative research, this may involve reviewing coded excerpts, comparing interpretations with the surrounding transcript, and documenting why particular labels were retained or revised.

In quantitative classification tasks, additional evaluation may involve examining error patterns and performance across relevant groups, where such analysis is methodologically justified and ethically permitted.

Researchers should avoid assuming that one universal fairness metric applies to every research design.

The NIST AI Risk Management Framework (2023) identifies harmful bias as an important consideration in trustworthy AI and emphasizes that risks must be understood within their contexts of use.

AI-generated labels should remain provisional until the researcher has established that they are suitable for the intended analysis.

How Should You Prompt AI to Identify Potentially Stigmatizing Interpretations?

A useful prompt asks AI to examine the evidential basis and possible consequences of proposed categories rather than simply remove all unfavorable terminology.

This prompt can assist with preliminary critical review. It does not establish that the resulting categories are unbiased or methodologically valid.

Researchers should also avoid submitting identifiable participant material to an unauthorized AI system for the purpose of conducting the review.

Do Not Remove Legitimate Findings Merely Because They Are Unfavorable

Ethical analysis does not require researchers to suppress every finding that reflects negatively on a practice, institution, or group.

For example, participants may report experiences of inadequate institutional support, unequal access to AI tools, or inconsistent implementation of policies.

If these findings are supported by the data, removing them simply because they may be uncomfortable would undermine the integrity of the research.

The distinction concerns whether the interpretation is adequately supported, proportionate, and presented without unnecessary stereotyping or generalization.

Researchers should preserve evidence-supported findings while avoiding language that attributes collective deficiencies beyond what the study can establish.

Ethical responsibility requires accurate representation, not automatic positivity.

Why Small Participant Groups May Face Particular Risks

Research involving small institutions or specialized professional communities may create additional identification concerns.

For example, a study may include only a few senior administrators responsible for institutional AI policy. Even without names, a report describing one subgroup as incompetent or resistant could make the individuals recognizable.

The risk depends on the information reported and the extent to which readers can connect the findings to particular people.

Researchers should consider whether reporting detailed group characteristics is necessary and whether less identifying descriptions would preserve the analytical contribution.

They should also avoid assuming that removing names automatically prevents deductive identification.

Who Is Responsible for AI-Generated Stigmatizing Findings?

Researchers remain responsible for the interpretations and claims they report, even when AI generated the initial labels.

For example, a researcher cannot adequately justify describing nonusers as technologically incompetent by stating that the classification was produced by ChatGPT.

The research team must evaluate whether the classification is supported by evidence and consistent with the methodological approach.

Where AI-generated outputs are used in consequential decisions, additional responsibilities may arise for institutions and other organizations involved in deploying the system.

Researchers should distinguish the model's role in generating suggestions from the human and institutional responsibility for adopting and using them.

When Should a Research Ethics Committee Be Consulted?

Institutional review may be appropriate when AI-generated classifications introduce new participant risks, involve sensitive characteristics, or are intended to influence decisions affecting individuals.

For example, a study originally approved to analyze instructors' experiences may later propose using AI to predict professional competence or policy compliance.

That change may introduce new analytical purposes and potential consequences beyond those considered in the original protocol.

Researchers should determine whether the proposed activity requires notification, amendment, or additional authorization.

More broadly, AI can change research ethical risks even when the original research questions remain unchanged.

When AI Labels Instructors as Resistant Without Sufficient Evidence

The revised interpretation does not require portraying the participant's decision as positive. It requires representing the stated reasoning accurately rather than imposing a generalized judgment.

The example illustrates how an apparently convenient AI-generated category can become ethically problematic when it attributes characteristics that the evidence does not establish.

Common Misconceptions About AI Discrimination and Stigmatization in Research

Examine What AI Labels Imply Before Treating Them as Findings

When AI assists with coding, classification, or interpretation, examine whether its categories accurately describe the evidence or introduce judgments about participants' motives, competence, or character.

Consider the potential consequences of those labels, especially when the study concerns identifiable individuals, small professional groups, or populations already subject to disadvantage.

Document consequential analytical decisions and examine the possible effects of reporting particular categories.

The objective is not to make every participant appear favorable. It is to ensure that the research describes people and groups according to defensible evidence rather than stereotypes introduced by an AI system.

Before Accepting AI-Generated Participant Classifications

Frequently Asked Questions About AI Discrimination and Stigmatization in Research

AI Can Classify Participants, but Classification Is Not Justification

How to Cite This Guide

This guide is intended to be read, shared, and used in research, teaching, and academic work. If you draw on its ideas, explanations, or other content, please acknowledge the source by citing the guide. Doing so gives appropriate credit and helps your readers locate the original resource.