Should Researchers Avoid AI When the Model Cannot Be Documented Adequately?

Can You Defend Research That Depends on an AI Model You Cannot Adequately Describe?

A researcher uses an online generative AI service to classify thousands of open-ended survey responses. The service produces useful categories, but its interface does not identify the precise model version. The provider may update the system without advance notice, and the researcher has not preserved the prompts or individual classification outputs.

When preparing the manuscript, the researcher realizes that another person may be unable to determine which model produced the results or how the classifications were generated.

Does this make the AI-assisted analysis scientifically unacceptable? Must researchers know every detail of a model's architecture and training data, or is a more limited record sufficient?

The issue is whether the available documentation allows the AI contribution to be understood, critically evaluated, and, where necessary, independently tested or reconstructed.

Avoid Consequential AI Use When Essential Documentation Cannot Be Preserved

What Adequate AI Model Documentation Means in Research

Documentation Is About Scientific Traceability, Not Merely Naming the Tool

A manuscript stating that "AI was used for analysis" provides little information about how the research was conducted.

Even identifying the provider and model name may be insufficient when the AI system performed a substantive analytical task. Readers may need to understand what information was supplied, how the model was instructed, how outputs were selected, and what researchers did with those outputs.

Scientific traceability means establishing a defensible connection between the original research material, the procedure performed, and the reported findings.

For example, when AI classifies interview excerpts, documentation should make it possible to understand the classification procedure and evaluate whether the reported categories are supported by the source material.

The purpose is not to create documentation for its own sake. It is to preserve the information necessary to scrutinize consequential research decisions.

What Information Should Researchers Record?

The appropriate documentation depends on the AI system and its function. For a generative AI workflow, the following categories may be relevant.

Not every item is relevant to every task. A spelling correction does not ordinarily require the same documentation as AI-assisted classification used to produce a primary outcome.

Researchers should also avoid retaining confidential prompts or outputs in insecure records. Documentation itself must comply with the project's information protection requirements.

Is Knowing the Exact Model Version Always Necessary?

Exact model identification is particularly valuable when model behavior may affect the research results.

A provider may offer a fixed version identifier, dated snapshot, or deployment reference. Other services expose only a product name, while the underlying model can change.

If a precise identifier is available, researchers should preserve it. If it is unavailable, they should document the information that can be established, including the product, access dates, relevant settings, and observed outputs.

Whether the remaining uncertainty is acceptable depends on the role of the model.

For example, a model used to suggest possible search terms may not require the same version precision as one used to generate the classifications from which a study's main findings are derived.

Researchers should never invent a model version or imply that a stable configuration existed when it cannot be established.

Do Researchers Need Access to Model Weights and Training Data?

Not universally. There is an important distinction between documenting an AI-assisted research procedure and fully reproducing the development of the AI model itself.

A researcher using an established commercial model for a limited supporting task may be able to document and validate the workflow without accessing proprietary model weights.

However, a study claiming to evaluate a model's internal mechanisms, training characteristics, or architecture may require information that the provider does not disclose.

The research question determines whether missing model-development information is a minor limitation or a fundamental obstacle.

What Do Model Cards and System Documentation Contribute?

Model cards are structured descriptions intended to communicate information about machine-learning models, including their intended uses, evaluation, limitations, and performance across relevant conditions.

Mitchell and colleagues introduced model cards as an approach to improving transparency in model reporting. Their framework emphasizes information that can help users understand the circumstances in which model performance was evaluated.

Provider documentation, system cards, technical reports, and model cards can support a researcher's assessment of suitability.

Nevertheless, a provider's documentation does not replace records of the researcher's own workflow. A model card may describe general performance without establishing how the system behaved on a particular research dataset.

Researchers should therefore combine available system-level information with task-specific documentation and validation.

Why Saving the Final Output Alone May Be Insufficient

An archived AI-generated result shows what was produced, but it may not explain how that result was obtained.

Suppose a researcher preserves a spreadsheet containing 2,000 AI-generated classifications but not the instructions used to create them.

Another researcher may be able to inspect the classifications, yet remain unable to determine whether the AI was asked to apply a predefined coding framework, invent categories, or revise labels based on earlier responses.

These differences matter because they can change the interpretation of the analytical procedure.

Where outputs materially influence findings, preserving relevant prompts, data preparation steps, and subsequent human decisions strengthens the audit trail.

What if the AI Service Does Not Preserve Conversation History?

Some systems offer limited interaction histories, temporary sessions, or interfaces that do not support reliable export.

Researchers should plan documentation before beginning consequential analysis rather than assuming that records can be recovered later.

Where permitted, they may maintain a structured research log, export outputs, preserve code, and record relevant configuration information.

For sensitive information, logging arrangements should be approved and protected appropriately.

If the tool cannot provide or support the records essential to the research design, a different system or procedure may be necessary.

Can a Black-Box AI System Be Used in Rigorous Research?

Yes, in some circumstances. Limited knowledge of a model's internal architecture does not automatically make every application invalid.

Researchers can sometimes evaluate a system through controlled inputs, reference datasets, external validation, error analysis, and comparisons with established procedures.

For example, a proprietary classifier might be evaluated against independently coded observations using prespecified performance measures.

However, black-box evaluation has limits. Strong performance on a test dataset does not establish that the system will behave consistently under all future conditions.

When the research question requires access to hidden mechanisms, or when essential behavior cannot be independently evaluated, the lack of transparency may be decisive.

How Does Documentation Relate to Reproducibility?

Documentation supports reproducibility, but the two concepts are not identical.

The National Academies of Sciences, Engineering, and Medicine distinguish computational reproducibility from replicability, emphasizing the importance of transparency about data and computational procedures.

For AI-assisted research, a complete record may allow others to understand and inspect what occurred even when the original system is no longer accessible.

However, documentation does not guarantee that rerunning the same prompt will produce an identical response. Model updates, nondeterministic generation, and unavailable infrastructure may limit exact reproduction.

Researchers should distinguish preserving evidence of the original procedure from demonstrating that the procedure can be rerun with sufficiently consistent results.

When Does Inadequate Documentation Become Unacceptable?

Documentation is inadequate when essential aspects of the research procedure cannot be established or evaluated.

For example, if AI-generated classifications form the primary outcome of a study but the researcher cannot identify the inputs, instructions, outputs, or validation process, the analysis may lack the traceability necessary to support its conclusions.

By contrast, incomplete model-version information may be manageable for a peripheral brainstorming task whose suggestions were subsequently evaluated independently.

The decision should therefore consider the scientific importance of the AI contribution, the consequences of uncertainty, and the availability of independent evidence.

What Should Researchers Do if Documentation Is Already Missing?

Begin by preserving whatever records remain available.

Possible sources include exported files, manuscript drafts, code repositories, timestamps, research logs, correspondence, and provider-generated identifiers.

Reconstruct the workflow only to the extent supported by evidence. Distinguish verified details from recollections or reasonable inferences.

If the original procedure cannot be established, researchers may need to repeat the analysis using a documented and validated method.

Where the undocumented output materially affects a submitted or published finding, the limitation may require editorial attention or correction.

Researchers should not manufacture missing prompts, outputs, or configuration details to create an appearance of reproducibility.

An AI-Assisted Classification Study Without a Recoverable Model Record

The original AI-generated labels may remain useful as exploratory material, but they should not be presented as a fully documented analytical procedure.

The central problem is not merely that the model was proprietary. It is that the researcher cannot adequately establish how the principal research results were produced.

Misconceptions About AI Model Documentation

Define the Documentation Requirements Before Using AI

Before incorporating AI into a consequential research procedure, determine which records would be necessary for another qualified researcher to understand and evaluate the work.

If those records cannot be preserved, reconsider the proposed system or modify the workflow.

The broader question of when researchers should avoid AI includes situations where inadequate documentation prevents scientific accountability.

Where model updates create additional uncertainty, examine whether changing AI models undermine the reproducibility required by the study.

Researchers should also establish which AI tool and model details can be documented and whether enough information can be preserved to reconstruct the workflow.

The defensible standard is documentation sufficient for the scientific purpose, not an unrealistic expectation that every AI system must be completely transparent internally.

Before Using an AI Model in a Consequential Research Procedure

Frequently Asked Questions About Undocumented AI Models

Use AI Only When Its Contribution Can Be Adequately Traced and Evaluated

How to Cite This Guide

This guide is intended to be read, shared, and used in research, teaching, and academic work. If you draw on its ideas, explanations, or other content, please acknowledge the source by citing the guide. Doing so gives appropriate credit and helps your readers locate the original resource.