03 · What You Need to Know
Situations in Which AI May Be Inappropriate for Research
When You Cannot Independently Verify Consequential AI Outputs
Generative AI can produce fluent explanations, plausible references, executable code, and apparently coherent interpretations. None of these characteristics establishes that an output is correct.
Researchers should avoid relying on AI when an error could materially affect their findings and they lack a credible means of detecting that error.
For example, asking AI to recommend a statistical model is not necessarily problematic. Accepting that recommendation without understanding its assumptions, checking its suitability, or validating the resulting analysis is substantially more difficult to defend.
The verification requirement should reflect the consequences of error. An inaccurate suggestion for a section heading has different implications from an incorrect calculation of a treatment effect.
When reliable verification is unavailable, the appropriate response may be to obtain qualified methodological assistance, use an established analytical procedure, or postpone the AI-assisted task.
When Confidential Research Information Cannot Be Protected
Research datasets may contain identifiable participant information, unpublished findings, commercially sensitive material, confidential peer-review manuscripts, or information protected by contractual agreements.
Entering such information into an AI system can create additional processing, storage, access, and disclosure risks. These risks depend on the provider's terms, technical controls, deployment arrangement, and institutional agreements.
Removing names does not necessarily make research information anonymous. Combinations of demographic characteristics, quotations, locations, or unusual experiences may still permit identification.
Researchers should not transfer restricted information to an AI service unless the intended processing is authorized and the relevant protections have been established.
Where an institution provides an approved, appropriately configured AI environment, some tasks may become permissible. Approval for one service should not be assumed to extend to another.
When the Tool's Data-Use Terms Are Unclear
Before entering research information into an AI system, researchers need to understand how the provider handles submitted material.
Relevant questions include whether inputs are retained, whether they may be used for model improvement, who can access them, what contractual protections apply, and whether deletion or retention controls are available.
A general statement that a platform is secure does not establish that it satisfies the requirements of a particular research project.
If the terms are unclear and the task requires restricted information, the responsible decision is to withhold the information until the uncertainty is resolved.
Researchers may still be able to test the tool with synthetic, public, or otherwise authorized material, provided that doing so does not create a misleading substitute for validation using the actual research context.
When You Do Not Understand the Method AI Is Recommending
AI can explain unfamiliar methods and suggest analytical alternatives, but it cannot transfer methodological competence simply by producing instructions.
A researcher who does not understand the assumptions of propensity score matching, structural equation modeling, multilevel analysis, or a machine-learning classifier may struggle to recognize inappropriate recommendations.
The problem is not unfamiliarity itself. Researchers regularly learn new methods during a project. The concern arises when an AI-generated procedure is implemented before the researcher has acquired sufficient understanding to evaluate its suitability.
Where a method is consequential, the researcher should be able to explain why it was selected, what assumptions it requires, how those assumptions were assessed, and how the findings should be interpreted.
If these questions cannot be answered, additional training, methodological consultation, or a more familiar valid approach may be preferable.
When Small Errors Could Cause Material Harm
Some research tasks have consequences beyond inaccurate academic reporting.
AI-assisted decisions may affect participant eligibility, risk classification, intervention allocation, safety monitoring, or interpretation of sensitive clinical and psychological information.
In these contexts, even an infrequent error may have serious consequences. Average model accuracy is insufficient if the system performs poorly for particular populations or fails in situations where mistakes are especially harmful.
The NIST AI Risk Management Framework emphasizes the importance of identifying, measuring, and managing AI risks within their intended contexts of use.
Researchers should avoid deploying an AI system in consequential participant-facing decisions unless its suitability has been established through appropriate validation, oversight, safeguards, and authorization.
Human review is not automatically sufficient. Reviewers must have the competence, information, authority, and opportunity to identify and correct errors before harm occurs.
When AI Use Would Violate an Applicable Rule or Approval
A research task may be technically feasible but impermissible under the conditions governing the project.
Restrictions may arise from institutional policies, research ethics approvals, informed consent, data-use agreements, funder requirements, copyright obligations, or journal instructions.
For example, confidential peer-review material should not be submitted to an unauthorized external AI service merely because the system could generate a useful review summary.
Likewise, a research protocol approved for one form of data processing does not necessarily authorize transferring participant information to an additional third-party provider.
Where permission is required, researchers should obtain it before proceeding. A later disclosure statement cannot retroactively make an unauthorized transfer permissible.
When the Model Cannot Be Documented Adequately
Some AI-assisted procedures require enough documentation for others to understand, audit, or reproduce how the research was conducted.
Relevant records may include the tool and model identifier, access date, system configuration, prompts, input preparation, output selection, verification procedures, and subsequent human modifications.
Not every task requires preservation of every interaction. However, when AI materially affects a reported analysis or research decision, insufficient documentation can prevent meaningful evaluation.
Researchers should reconsider AI use if the system changes unpredictably, essential configuration information is unavailable, or the workflow cannot preserve the records necessary to substantiate its results.
The required documentation should reflect the methodological significance of the contribution rather than an arbitrary demand to archive every interaction.
When Model Changes Threaten Essential Reproducibility
Generative AI systems may change through model updates, modified safety settings, altered retrieval mechanisms, or changes to the provider's infrastructure.
Even when a prompt is preserved, a later interaction may produce a different response. Identical prompts do not necessarily guarantee identical outputs, including when the model identifier appears unchanged.
This variability matters when a research procedure depends on consistent classification, extraction, scoring, or interpretation.
Where exact or sufficiently stable reproduction is essential, researchers should evaluate whether a fixed model version, controlled environment, archived outputs, validated deterministic procedure, or conventional software would better support the study.
AI may remain useful for exploratory work, but an unstable model should not be treated as a reproducible measurement instrument without adequate validation and documentation.
When a Simpler Validated Tool Can Perform the Task More Reliably
Generative AI is not necessarily the most appropriate tool for a task simply because it can produce an answer.
Calculating descriptive statistics, applying a predefined coding rule, validating a data format, or extracting structured fields may be more reliably performed through conventional software or deterministic procedures.
For instance, a spreadsheet formula or a validated statistical package can calculate a median without introducing the variability associated with a conversational model's generated response.
This does not mean conventional tools are error-free. Their assumptions, implementation, and user inputs still require checking.
The relevant comparison concerns reliability, auditability, efficiency, and the demands of the specific task. Adding AI to an otherwise straightforward workflow may introduce dependencies and verification burdens without improving the scientific result.
When AI Saves Time but Adds No Scientific Value
Time savings can be legitimate, particularly for repetitive or administrative work. However, efficiency should not be confused with improvement in research quality.
An AI tool may produce a literature summary quickly while requiring extensive checking for missing qualifications, inaccurate citations, or distorted interpretations. The apparent time saved during generation may be offset by the work required to validate the output.
Similarly, AI-generated analysis code may be less efficient overall if the researcher spends more time debugging it than would have been required to implement a familiar procedure.
The decision should consider the full workflow, including preparation, verification, correction, documentation, and potential consequences of error.
AI is not necessarily inappropriate whenever its benefit is limited to speed. But where it introduces substantial additional risk or complexity without a meaningful compensating benefit, avoiding it may be the better choice.
When AI Would Replace a Judgment the Researcher Must Personally Make
Scientific judgment involves selecting defensible research questions, evaluating evidence, interpreting uncertainty, and accepting responsibility for methodological decisions.
AI can challenge assumptions, identify possible alternatives, and provide explanations. These contributions may improve deliberation when independently evaluated.
The problem arises when researchers treat an AI recommendation as authoritative without examining its evidentiary basis.
For example, an AI system may recommend excluding observations because they appear anomalous. The researcher must still determine whether exclusion is justified by the study design, prespecified criteria, measurement characteristics, and relevant methodological standards.
If the researcher cannot defend the decision without appealing to the AI system's authority, the workflow has not provided meaningful scientific oversight.
When the Research Task Is Not Suitable for Generative AI
Some tasks require precise operations that a general-purpose language model is not designed or validated to perform reliably.
Examples may include exact numerical computation, exhaustive database retrieval, strict application of formal classification rules, or analysis requiring guaranteed preservation of source text.
Specialized AI systems may be appropriate for some of these tasks when validated for the intended purpose. The limitation should therefore be attributed to the particular system and workflow rather than to every technology described as AI.
Researchers should match the tool to the task, not redefine the task around whichever AI tool is most accessible.
