03 · What You Need to Know
What Makes Generative AI Safeguards Different?
The Starting Point Is Not “AI Is Dangerous”
Generative AI should not be treated as uniquely suspect simply because it is artificial intelligence. Conventional research tools have failure modes too, and human researchers themselves make mistakes.
The appropriate comparison is functional.
As explained when comparing generative AI with other research software, safeguards should depend on how a system produces its output, what that output represents, and how consequential an undetected error would be.
A grammar correction, generated statistical interpretation, synthetic image, classification of participant data, and invented citation do not create identical risks merely because all five involve AI.
Generative AI Can Produce False Information That Looks Complete
One of the clearest reasons for additional safeguards is the ability of generative systems to produce false or unsupported information in a fluent and convincing form.
NIST's Generative Artificial Intelligence Profile identifies confabulation as a generative-AI risk. Such outputs can include erroneous facts, false logic, or fabricated citations presented confidently enough that users may treat them as reliable.
This differs from many conventional software failures because the system can create the informational object itself.
A conventional reference manager may contain an incorrect year because bad metadata were imported. A generative model may create an entire article title, author list, journal name, volume, page range, and DOI-like identifier for a publication that never existed.
Watch Out
Do not use the generative system's confidence, specificity, formatting, or self-assurance as evidence that a factual claim is true. Consequential claims require verification against an appropriate external source.
Verification Must Be Independent When the System Generated the Claim
If an AI system tells you that a paper exists and you ask the same system, “Are you certain?”, you have not independently verified the reference.
The same problem arises when researchers ask a model to check whether its own statistical explanation, quotation, legal claim, methodological recommendation, or factual statement is correct.
The system may reconsider and correct itself. That can be useful. But authoritative verification requires an independent reference point appropriate to the claim.
| Generated output |
Better verification source |
| Journal article citation |
The actual article, publisher, DOI registration metadata, or appropriate bibliographic database |
| Statistical explanation |
The data, diagnostics, model, calculations, and authoritative methodological literature |
| Software syntax |
Execution, tests, and official software or package documentation |
| Institutional or publisher policy |
The current official policy of the responsible organization |
| Claim about a supplied paper |
The relevant passage, table, figure, or analysis in the paper itself |
This principle is especially important because large language models generate responses rather than inherently retrieving verified answers.
Generative AI Can Create New Scholarly Content
Traditional research software often transforms information according to relatively bounded operations. Generative AI can instead create new prose, code, images, interpretations, summaries, classifications, methodological suggestions, and other content.
That introduces questions that do not arise in the same way when software merely performs a predefined calculation.
Where did the substantive claim originate? Does the generated interpretation follow from the evidence? Is the generated code methodologically appropriate? Did the system introduce an argument that the researcher had not developed? Is disclosure required? Does generated material reproduce protected content or create attribution concerns?
The more substantive the generation becomes, the more important it is to distinguish AI assistance from AI-generated scholarly material.
Source Provenance Can Be Unclear
Researchers are accustomed to asking where information comes from.
A database record has a source. A paper has authors and a publication venue. A dataset has provenance. A quotation should be traceable to a document.
A generated sentence may not have an equivalent one-to-one source.
An LLM's output is generated from learned patterns, current context, system instructions, and potentially retrieved information or tool outputs. Even when the resulting statement is factually correct, researchers may not know which underlying source establishes it unless the system provides genuine retrieval and the source is inspected.
NIST's discussion of information integrity emphasizes information that can be linked to original sources with appropriate evidence, distinguished from inference or opinion, and verified or authenticated.
For research, that means provenance should become more important as a generated statement moves closer to the evidential core of the study.
Fluency Creates a Risk of Automation Bias
Bad output does not always look bad.
Generative AI can produce coherent, polite, technically sophisticated responses even when they contain important errors. NIST specifically warns that increasing reliability and complexity can encourage excessive deference to generative systems, a form of automation bias.
This creates an unusual human-factors problem. Researchers may lower their skepticism precisely because the system appears competent.
A safeguard therefore needs to address not only model error but researcher behavior.
Useful practices include deliberately checking high-consequence outputs, asking what evidence would falsify the generated answer, comparing against independent sources, and avoiding the assumption that confidence or detail indicates correctness.
Natural Language Can Conceal How Much Work the System Is Actually Doing
Traditional software often exposes its operation more clearly. You choose a statistical model, write a formula, select a database filter, or execute a function.
With generative AI, a short instruction can trigger a much broader transformation.
“Improve this paragraph” might alter grammar only, or it might introduce new claims. “Analyze these interviews” might classify passages, infer themes, summarize participants, and generate interpretations. “Fix this code” might rewrite a substantial portion of the analysis.
The convenience of natural-language interaction therefore creates a safeguard requirement: researchers need to inspect what changed rather than infer the scope of the operation from the simplicity of the prompt.
Data Can Leave the Research Environment
Many traditional research workflows can operate entirely on local infrastructure. A dataset opened in locally installed statistical software does not necessarily leave the researcher's computer.
Generative AI services may operate through external infrastructure. Prompts, files, images, or other supplied information may be processed according to technical and contractual arrangements that differ among services, deployments, account types, and institutions.
This becomes consequential for:
- identifiable or potentially identifiable participant information;
- confidential research records;
- unpublished findings;
- proprietary or commercially sensitive material;
- embargoed data;
- peer-review manuscripts or grant proposals;
- copyrighted or licensed content subject to restrictions.
UNESCO's guidance emphasizes data privacy and human-centred governance, while the European Commission's living guidelines address responsible information handling in research uses of generative AI. The 2026 update also adds attention to AI interactions involving third parties and information-management environments.
The safeguard must occur before the upload. Asking the AI afterward whether the information was confidential is somewhat late in the methodological proceedings.
AI Can Introduce Third-Party Risks Researchers Do Not See
AI may enter a research workflow without the researcher directly opening a chatbot.
Meeting software may generate transcripts or summaries. A collaborator may use AI to process shared material. An information-management system may contain embedded AI features. A service provider may introduce AI into a workflow.
The European Commission's 2026 revision specifically highlights interactions with third parties using AI during meetings or information management and draws attention to risks such as instructions hidden from human oversight.
This expands the safeguard question from “Am I using AI?” to “Where might AI be processing or influencing research information in this workflow?”
Variable Outputs Can Complicate Reproducibility
Some generative systems may produce different responses to identical or similar prompts. Behavior can also change when the model, application, retrieval system, system instructions, or underlying service changes.
This does not make reproducible research impossible. It means documentation may need to capture the elements that materially affect the result.
Depending on the task, that could include:
- the system or model used;
- the date or relevant version;
- the prompt or instruction;
- important settings;
- documents or data supplied;
- retrieval sources;
- generated outputs retained for analysis;
- human review and modification procedures.
Not every grammar correction requires an archival dossier. Documentation should be proportionate to the methodological importance of the AI contribution.
Bias Can Enter Through More Than the Researcher's Dataset
Researchers already consider bias in sampling, measurement, analysis, and interpretation.
Generative AI can introduce another source because model behavior reflects training data, design decisions, adaptation procedures, system instructions, and deployment context.
A system may represent some languages, cultural contexts, populations, scholarly traditions, or conceptual frameworks better than others. Generated classifications or interpretations may therefore reproduce patterns that are not visible from the immediate research dataset.
UNESCO's human-centred approach explicitly raises concerns involving inclusion, equity, linguistic and cultural diversity, privacy, and human agency.
For consequential classification or interpretation, researchers should therefore investigate whether performance or meaning changes across relevant groups or contexts rather than assuming one global level of reliability.
AI Can Encourage Researchers to Operate Beyond Their Expertise
Generative AI lowers technical barriers. That is one of its most valuable properties.
A researcher can generate Python code without knowing much Python, request a sophisticated statistical model without having implemented one before, or obtain terminology from an unfamiliar discipline.
The corresponding risk is an evaluation gap: AI may make it easier to produce an output than to determine whether the output is correct.
A safeguard therefore needs to include competence. If the research depends on an output, someone involved in the research should possess enough relevant expertise to evaluate it properly.
This is why meaningful human oversight cannot be reduced to a person clicking “accept.”
Safeguards Should Increase With Consequence
Not every use of generative AI deserves the same controls.
| Example use |
Typical consequence if wrong |
Safeguard emphasis |
| Suggesting alternative titles |
Usually low |
Accuracy and fit |
| Editing researcher-written prose |
Low to moderate |
Preservation of meaning and claims |
| Summarizing literature |
Moderate to high |
Source grounding and direct verification |
| Generating analysis code |
Potentially high |
Testing, methodological validation, documentation |
| Classifying primary research data |
Potentially high |
Performance validation, bias assessment, data governance |
| Interpreting findings |
High |
Evidence, theory, expertise, alternative explanations, human judgment |
| Processing sensitive participant information |
Potentially severe |
Privacy, security, authorization, ethics, contractual and institutional compliance |
The principle is not “more AI means more danger.” It is that stronger safeguards become appropriate when the combination of uncertainty and consequence becomes more serious.
Traditional Research Safeguards Still Apply
Generative-AI safeguards should be added to ordinary research practice, not substituted for it.
A researcher can verify every AI-generated sentence and still conduct a badly sampled study. Perfect AI disclosure cannot repair invalid measurement. Secure data handling does not make an inappropriate statistical model appropriate.
The European Commission's current AI-in-science approach explicitly combines AI adoption with preserving scientific integrity and methodological rigor.
Generative AI introduces additional failure points. It does not repeal the old ones.
