03 · What You Need to Know
Evaluating Research That Could Enable Harmful Uses of Information and Technology
Why Do These Risks Arise in Otherwise Legitimate Research?
Many research methods generate information or capabilities that can serve different institutional objectives.
Predictive models can identify individuals who may benefit from assistance, but they can also support exclusionary classification. Monitoring systems can improve operational safety while creating opportunities for intrusive observation. Behavioral research can inform effective instruction while also supporting techniques intended to influence people without adequate transparency.
The research method itself does not always determine which application will follow.
Its consequences may depend on who controls the technology, which decisions it informs, what incentives influence its use, and whether affected individuals can understand or contest the resulting actions.
Researchers should therefore evaluate the relationship between technical capability and institutional context rather than assume that beneficial intentions guarantee beneficial outcomes.
How Do Discrimination, Surveillance, Manipulation, and Exploitation Differ?
Discrimination
Unjustified differential treatment or disadvantage associated with protected characteristics or other relevant group distinctions, depending on the applicable ethical and legal framework.
Surveillance
Systematic observation or monitoring of individuals or groups, which may become intrusive or coercive depending on its purpose, scope, and governance.
Manipulation
Attempts to influence decisions or behavior through methods that may undermine informed, autonomous judgment, particularly when important information is concealed.
Exploitation
Taking unfair advantage of individuals or groups, often through unequal power, limited alternatives, or the extraction of benefits without appropriate regard for their interests.
These concepts overlap but should not be treated as synonyms. Monitoring is not necessarily discriminatory, and behavioral influence is not automatically manipulation. The ethical significance depends on the specific practices and consequences.
How Can Research Enable Discrimination?
Research may enable discrimination when its findings, classifications, or technologies are used to support unjustified differences in treatment.
For example, a predictive model developed from historical student records may identify patterns associated with academic performance. If the institution subsequently uses those predictions to restrict educational opportunities, existing inequalities may influence who receives access.
However, an observed statistical difference between groups does not automatically establish unlawful discrimination or algorithmic bias.
Researchers should examine measurement validity, population representation, error rates, intended decisions, and the consequences of incorrect classifications.
They should also distinguish predictive accuracy from the ethical legitimacy of using predictions to make consequential decisions.
A technically accurate model does not independently establish that its deployment is fair.
Can a Model Be Accurate and Still Produce Discriminatory Outcomes?
Yes. Predictive accuracy and fairness are related but distinct considerations.
A model may produce statistically accurate predictions while being used for decisions that impose disproportionate burdens on particular groups.
For example, a model predicting educational disengagement may reflect unequal access to connectivity, devices, and instructional support.
Using those predictions to deny opportunities could reinforce the conditions associated with the predicted outcome.
However, fairness cannot always be established using one universal statistical metric. Different measures may reflect competing assumptions about acceptable error and equitable treatment.
Researchers should examine both technical performance and the institutional decisions the model is intended to support.
When Does Research-Enabled Monitoring Become Problematic Surveillance?
Monitoring may be appropriate when it serves a legitimate purpose and operates under proportionate safeguards.
For example, recording learning management system activity may help researchers understand how students engage with course materials.
However, the same records could support continuous observation of students' behavior beyond what is necessary for the educational purpose.
Concerns may increase when individuals have limited ability to refuse monitoring, do not understand what is collected, or face consequential decisions based on incomplete behavioral indicators.
Researchers should examine necessity, proportionality, transparency, access, retention, and the consequences of monitoring.
Consent alone may not resolve every concern, particularly where institutional power limits meaningful choice.
How Can Behavioral Research Enable Manipulation?
Research into motivation, persuasion, attention, or decision-making can support legitimate educational and public-interest interventions.
For example, reminders may help students complete assignments, and appropriately designed feedback may support self-regulated learning.
However, behavioral techniques may become ethically problematic when they intentionally exploit vulnerabilities, conceal material information, or undermine individuals' ability to make informed choices.
The distinction depends on the intervention's purpose, transparency, consequences, and the degree of autonomy retained by those affected.
Researchers should avoid assuming that every attempt to influence behavior is manipulation. Teaching itself involves influencing knowledge and behavior.
The ethical question is whether the influence respects legitimate educational purposes and the agency of the individuals involved.
What Does Exploitation Look Like in Research Applications?
Exploitation may occur when research capabilities are used to obtain benefits from individuals or communities under unfair conditions.
For example, an organization might use detailed behavioral information to identify students who are particularly susceptible to commercial pressure and target them with offers that exploit financial insecurity.
Another concern arises when communities provide extensive data while receiving little consideration in decisions about how the resulting technologies are commercialized.
However, unequal distribution of benefits does not automatically establish exploitation.
Researchers should examine consent, bargaining power, reasonable expectations, the distribution of burdens and benefits, and the actual uses of the resulting knowledge.
Which Research Capabilities Deserve Particular Attention?
| Research capability |
Potential beneficial application |
Potential harmful application |
| Student risk prediction |
Timely academic assistance. |
Exclusionary profiling or punitive classification. |
| Behavioral monitoring |
Understanding learning engagement. |
Disproportionate or covert surveillance. |
| Emotion or affect inference |
Investigating adaptive interfaces. |
Unsupported judgments about mental states or suitability. |
| Personalized persuasion |
Supporting beneficial educational choices. |
Exploiting vulnerabilities or reducing informed autonomy. |
| Automated identity analysis |
Supporting appropriate verification processes. |
Unauthorized tracking or discriminatory identification. |
| Community-level analytics |
Identifying inequalities in educational access. |
Stigmatization or unfair resource allocation. |
These examples illustrate competing applications. They do not establish that every project developing these capabilities will produce the listed harms.
Why Does Institutional Power Matter?
Research applications do not occur in a social vacuum. The authority of the organization using a technology can affect the consequences for individuals.
A student may be able to decline an optional educational application but have little practical ability to refuse monitoring required by a university.
Similarly, an employee may face consequences for declining participation in an employer-controlled system.
Researchers should therefore examine whether consent is meaningful, whether alternatives exist, and whether affected individuals can challenge decisions.
Power asymmetry does not automatically make every institutional research activity unethical, but it may increase the importance of independent safeguards.
Does Participant Consent Resolve These Risks?
No. Consent addresses important aspects of research participation, but it does not necessarily authorize every future application of the resulting knowledge.
Participants may consent to providing educational records for research without agreeing that a model developed from those records should determine future admissions or disciplinary decisions.
Moreover, downstream applications may affect individuals who never participated in the original study.
Researchers should distinguish authorization to collect and analyze data from the ethical justification for deploying a resulting system.
The latter may require separate evaluation of fairness, validity, transparency, and institutional accountability.
How Can Researchers Identify Foreseeable Misuse?
Researchers should examine the capability produced, likely users, accessibility, deployment incentives, and potential consequences.
A harmful application may be more foreseeable when it requires little modification of the original technology or resembles practices already occurring in the intended environment.
For example, a student classification model developed for administrative decision support may have a relatively direct pathway to exclusionary use.
By contrast, a speculative application requiring several unlikely technical developments may provide a weaker basis for imposing restrictions.
Risk assessment should distinguish credible scenarios from conjecture and document uncertainty where evidence is incomplete.
What Role Does the NIST AI Risk Management Framework Play?
The National Institute of Standards and Technology's AI Risk Management Framework provides a voluntary framework for managing risks associated with AI systems.
Its functions, Govern, Map, Measure, and Manage, support structured consideration of organizational responsibilities, context, risk assessment, and responses.
Researchers may draw on these principles when examining potential discrimination, harmful deployment, or inadequate oversight.
However, the framework is not a universal legal requirement or a validated ethics approval instrument for every research study.
It should be applied according to the actual project and relevant institutional obligations.
How Does UNESCO Address These Concerns?
UNESCO's Recommendation on the Ethics of Artificial Intelligence addresses human rights, dignity, fairness, privacy, transparency, and broader social consequences of AI.
Its principles provide a policy reference for considering the effects of AI systems on individuals and communities.
However, the Recommendation should not be represented as a directly enforceable domestic statute in every country.
Researchers should distinguish international ethical recommendations from binding legal requirements that apply in a particular jurisdiction.
Its value lies partly in encouraging consideration of social and institutional consequences beyond technical performance.
Should Researchers Test for Discriminatory Effects?
Where research develops a model intended for consequential classification, examining differential performance and potential harms across relevant populations may be appropriate.
However, subgroup analysis requires sufficient data, suitable measurement, and careful interpretation.
Small subgroup samples can produce unstable estimates, and fairness assessments may involve competing criteria.
Researchers should not claim that a model is fair merely because one metric shows no statistically significant difference.
They should describe the limitations of the available evidence and consider whether deployment requires further evaluation.
Can Human Oversight Prevent Harm?
Human review may provide opportunities to challenge automated recommendations and consider contextual information.
However, adding a human decision-maker does not automatically eliminate bias or inappropriate reliance on automated outputs.
Reviewers may defer excessively to model predictions or lack the information necessary to question them.
Researchers should examine whether oversight personnel possess appropriate authority, expertise, and opportunities to correct errors.
Meaningful oversight requires more than including a human approval step in a workflow.
Should Researchers Restrict Access to Potentially Harmful Technologies?
Sometimes. Access restrictions may be appropriate when releasing particular models, datasets, or operational tools would materially increase serious foreseeable risks.
However, restrictions can also reduce reproducibility, independent scrutiny, and beneficial applications.
Researchers should identify the specific harmful capability and consider whether narrower safeguards can address it.
For example, sharing analytical documentation while restricting identifiable student-level records may preserve important opportunities for evaluation.
The broader issues concerning scientific openness and responsible research are relevant when disclosure creates competing ethical obligations.
When Should the Research Question Be Changed?
Changing the question may be appropriate when serious foreseeable harm is closely connected to the intended capability and cannot be adequately addressed through procedural safeguards.
For example, researchers might reconsider a project intended to generate individual suspicion scores for students when the scientific objective could instead be pursued through studying conditions that support academic integrity.
However, a revised question may produce different knowledge and should not be treated as a cost-free substitute.
Researchers should compare the scientific value, anticipated harms, and methodological implications of available alternatives.
The criteria discussed in deciding whether misuse risks should change a research question provide a related decision framework.
Can Accurate Research Still Be Used to Justify Harm?
Yes. Empirically valid findings may be applied beyond the population, context, or decision for which they were established.
A model that predicts student performance within one institution may not be suitable for evaluating applicants from another population.
Even where predictions are accurate, their use for exclusionary decisions may require ethical justification beyond statistical performance.
Researchers should distinguish what the evidence establishes from what institutional users wish to infer.
The possibility of accurate findings causing unfair population harm is especially relevant when research informs consequential policy.
Are Researchers Responsible for Every Harmful Application?
No. Researchers cannot necessarily control how independent organizations use published findings.
However, responsibility may increase when researchers directly participate in deployment, know of foreseeable risks, or possess meaningful influence over safeguards.
For example, a researcher implementing an automated student classification system may be able to establish validation requirements and contestation procedures.
A researcher publishing a general theoretical finding may have substantially less control.
The appropriate responsibility depends on foreseeability, conduct, influence, and applicable obligations.
What if a Sponsor Wants to Use the Findings for Harmful Purposes?
Researchers should examine whether the proposed application is supported by the evidence and whether it creates serious ethical or legal concerns.
For example, a sponsor may seek to use a predictive model to exclude students rather than offer academic assistance.
Possible responses include clarifying limitations, documenting concerns, seeking institutional advice, or declining further involvement where appropriate.
However, researchers should not assume that disagreement over policy automatically establishes misconduct by the sponsor.
The distinction between empirical validity and ethical application should remain explicit.
How Can Researchers Document Their Assessment?
Researchers should record the intended capability, plausible harmful applications, relevant evidence, affected populations, safeguards, and unresolved uncertainties.
The record should explain why the selected design or dissemination strategy is proportionate to the identified risks.
Where consultation informs the decision, the relevant advice and its implications should be documented.
Researchers should avoid presenting subjective judgments as validated quantitative risk scores without appropriate methodological support.
Documentation should also identify the limits of the researcher's control over downstream applications.
What Is the Most Defensible Approach?
The most defensible approach begins with the actual capability rather than the researcher's intentions alone.
Researchers should examine who may use the capability, which decisions it could influence, and whether foreseeable harms can be reduced through appropriate design and governance.
Not every predictive or monitoring technology is unethical. Equally, technical validity does not automatically justify every deployment.
The goal is to support legitimate scientific work while avoiding unnecessary creation or distribution of capabilities that facilitate serious harm.
Watch Out
Do not assume that accurate predictions, informed consent, or institutional approval automatically make a consequential technology ethically acceptable. Examine who controls the system, how its outputs may be used, whether affected individuals can challenge decisions, and whether the resulting harms are reasonably foreseeable.
