Should Researchers Treat an AI Model Version Like a Software Version?

Should an AI Model Version Be Recorded Like Python, R, or a Statistical Package?

Researchers routinely document the software used to analyze their data. A methods section might identify an R version, a Python package, or a statistical analysis application because software updates can change how calculations are performed.

But what happens when part of the analysis is performed by a generative AI model?

Suppose a researcher reports using ChatGPT to classify interview excerpts. Another researcher attempts to repeat the procedure a year later, using the same prompt but a newer model. The classifications differ. The original study identifies the AI service but not the specific model version.

Was the original methodological documentation sufficient? And should researchers treat an AI model as another software dependency, or does generative AI require a different approach to version control?

Yes, AI Model Versions Should Be Documented as Research Dependencies

Why AI Model Versions Matter in Research Methodology

A Model Version Can Influence the Results Just as a Software Version Can

Research software is not merely a convenient interface for conducting an analysis. Its algorithms and implementation details can influence the results.

For example, changes to statistical packages may affect numerical precision, optimization procedures, default settings, or how particular functions handle missing data.

Generative AI models present a comparable concern. Different model versions may interpret prompts differently, produce different classifications, extract different information, or respond differently to ambiguous research material.

When AI generates variables, codes observations, or contributes to analytical decisions, the model becomes part of the method used to produce the evidence.

Research on computational reproducibility has emphasized the importance of preserving model information, data, code, and computational workflows. Heil and colleagues, writing in Nature Methods, proposed reproducibility standards for machine learning in the life sciences that include model and code publication alongside sound computational practices.

Although the requirements for generative AI research may differ, the underlying methodological principle is relevant: a research result cannot be fully understood without identifying the computational procedure that produced it.

What Exactly Counts as an AI Model Version?

The phrase "model version" can refer to several different identifiers. These should not be treated as interchangeable.

A researcher who reports only the provider name has documented considerably less than someone who records a specific model snapshot and its configuration.

Nevertheless, the most precise identifier available may depend on the service. Researchers should report what they can verify rather than invent a version number or assume that a model family name represents an immutable release.

Why AI Model Versioning Is More Complicated Than Conventional Software Versioning

Conventional software packages often have explicit release numbers, source repositories, changelogs, and installation mechanisms. Researchers may be able to reinstall a particular version or preserve it within a computational environment.

AI models can share some of these characteristics, especially when their weights and associated files are distributed through version-controlled repositories.

However, proprietary AI services may expose only a model identifier and a remote interface. Researchers may not have access to the model weights, tokenizer implementation, serving software, or provider-controlled instructions.

The distinction becomes important when a service changes. A software version number may refer to an archived package that can still be installed, while a hosted AI model identifier may refer to a system that the provider can modify or discontinue.

Version identification is often feasible even for hosted services. Version preservation may not be possible without access to the underlying artifacts.

Why Researchers Should Distinguish Fixed Versions From Moving Aliases

Some model providers expose identifiers that refer to particular releases. Others offer aliases that point to whichever version the provider currently designates.

For example, a hypothetical model repository could have a branch named main and a specific commit hash identifying the exact files used in an analysis.

If the researcher loads the model from main on two different dates, the repository contents may have changed. Loading the files from a fixed commit provides a stronger basis for reconstructing the original model artifacts.

Hugging Face's model-sharing documentation recommends using explicit revisions, including commit hashes, when reproducibility across executions matters. Its documentation on model revisions explains how a particular repository state can be selected.

Hosted AI APIs introduce a similar concern when a model identifier functions as a moving alias.

Researchers should consult official provider documentation to establish whether a model identifier is fixed, versioned, or subject to reassignment.

What Information Should Accompany the Model Version?

Recording the model version is necessary for many AI-assisted analytical tasks, but it is rarely sufficient on its own.

A useful methodological record should connect the model identity to the actual conditions under which it was used.

Not every AI interface exposes all these details. If the model version or a configuration setting cannot be obtained, researchers should acknowledge that limitation rather than supply an estimated value.

The technical settings deserve separate attention because generation parameters can affect the reproducibility of model outputs even when the model identifier remains unchanged.

Does the Model Version Need to Appear in the Published Methodology?

When AI contributes directly to research data, measurement, analysis, or interpretation, identifying the model and relevant execution conditions is generally appropriate for the methods section or supplementary methodological documentation.

The necessary detail depends on the research task and applicable journal, institutional, or funder requirements.

A study using AI to extract numerical information from research articles may require a more detailed record than a manuscript that used AI only to improve sentence structure.

For consequential analytical uses, a methodology might describe the provider, model identifier, access method, execution dates, prompts, configuration, validation procedure, and handling of errors.

If the model has changed during the study, the researcher should identify which observations or analytical stages used each version.

This is particularly important when data generated under different model versions are combined. The change may introduce systematic differences that could be mistaken for variation in the underlying research material.

What If the AI Model Is Updated During an Ongoing Study?

Researchers should treat a model update as a potential change in analytical conditions rather than assume the new version is interchangeable with the previous one.

For example, an AI-assisted content analysis may begin with one model version and continue after the provider introduces a replacement.

If the newer version interprets the coding instructions differently, the resulting dataset could contain a version-related measurement difference.

A reasonable response is to evaluate both versions on a common reference sample, compare their outputs, and determine whether the differences are scientifically consequential.

The researcher should document the change and its implications rather than silently combine results from the two systems.

The broader consequences of AI model updates for research reproducibility become particularly serious when the earlier version is no longer accessible.

Does Recording a Model Version Guarantee Reproducibility?

No. A model identifier documents an important dependency, but exact reproduction may require additional conditions to remain constant.

PyTorch's reproducibility documentation explains that identical random seeds do not necessarily guarantee identical results across software releases, platforms, or CPU and GPU executions. Deterministic algorithm settings can reduce certain sources of variability, but they are not sufficient for every application. The official reproducibility documentation discusses these limitations.

Generative AI systems introduce further complications through decoding configurations, hidden service behavior, and changing contextual inputs.

Consequently, researchers should distinguish documenting the model from demonstrating that repeated executions produce the same results.

For substantive analytical tasks, the strongest evidence usually combines version documentation with archived outputs, independent validation, and an assessment of output stability.

Documenting an AI Model Used to Classify Research Abstracts

The hypothetical disagreement rate is 12%, calculated as 12 disagreements divided by 100 abstracts. This figure does not independently establish that either model is more accurate or that the two versions are unsuitable for every research purpose.

Its significance depends on the category definitions, the pattern of disagreement, and the consequences for the study's findings.

Without version documentation, the researcher might mistakenly treat all 600 classifications as products of one unchanged measurement procedure.

Common Mistakes in Documenting AI Model Versions

How to Document AI Model Versions in Your Research

The appropriate level of documentation should reflect how consequential the model's outputs are to the study.

For AI-assisted writing support, recording the tool and complying with applicable disclosure policies may be sufficient. For AI-generated classifications, extracted measurements, or analytical interpretations, more detailed version documentation is warranted.

Maintain a Model Record Alongside the Analysis

For substantial AI-assisted workflows, a simple model record can make later verification considerably easier.

It might include the model identifier, execution dates, API or software versions, prompts, settings, input dataset revision, output archive location, and a description of any changes made during the study.

Researchers can store this information in a structured metadata file, analysis log, or supplementary methods document. Version control can help preserve the history of changes to these records.

Where model outputs contribute directly to reported findings, archiving the original outputs provides evidence that a model identifier alone cannot supply.

The objective is not to reproduce every technical detail of the provider's infrastructure in a journal article. It is to preserve enough information for another researcher to identify the computational dependency, understand the analytical procedure, and evaluate the reproducibility of the reported results.

AI Model Version Documentation Checklist

Frequently Asked Questions About AI Model Versioning

AI Model Versions Belong in the Research Record

How to Cite This Guide

This guide is intended to be read, shared, and used in research, teaching, and academic work. If you draw on its ideas, explanations, or other content, please acknowledge the source by citing the guide. Doing so gives appropriate credit and helps your readers locate the original resource.