Should Researchers Record the Exact AI Model Version Being Studied?

Is Naming the AI Model Enough for Someone Else to Reproduce a Study?

A researcher evaluates a commercial language model's ability to generate accurate explanations of research methodology concepts.

The methods section states that the study used ChatGPT. The researcher includes the questions, evaluation rubric, and results but does not identify the underlying model or testing dates.

Several months later, another researcher attempts to replicate the evaluation. The same questions produce different answers.

Did the model improve? Was a different model used? Did the provider update the service? Or did the responses vary because of ordinary generation behavior?

Without more precise documentation, these explanations may be difficult to distinguish.

The problem raises an important methodological question: how precisely must researchers identify an AI system for their findings to be interpretable, auditable, and reproducible?

Record the Most Specific Model Version Available and Acknowledge What Cannot Be Identified

Why Model Version Identification Matters for Scientific Research

What is an AI model version?

An AI model version identifies a particular release, revision, checkpoint, or deployment of an artificial intelligence system.

Depending on the provider, versions may be represented through dated identifiers, snapshot names, revision numbers, model-file hashes, or other technical metadata.

However, terminology is not consistent across providers.

A model name may identify a family of related systems rather than one immutable set of model weights.

Similarly, a service endpoint may retain the same name while the underlying implementation changes.

Researchers should therefore distinguish the displayed model name from an exact model snapshot whenever that distinction is relevant.

Why is recording the model version scientifically important?

Model identification establishes which computational system produced the observations or assisted with a consequential research procedure.

For example, a study examining citation hallucinations may obtain different results from different versions of the same model family.

Without adequate identification, another researcher may unknowingly evaluate a different system while attempting replication.

Version documentation also helps readers determine whether findings remain applicable to currently deployed systems.

However, the scientific value of version information depends on its accuracy and specificity.

Recording an unsupported version label can be more misleading than explicitly acknowledging that the exact version was unavailable.

What information should researchers distinguish?

These details are complementary. A precise model identifier may still be insufficient if consequential deployment settings or testing conditions are omitted.

Is writing ChatGPT in the methods section sufficient?

Usually not when the research evaluates model behavior or relies materially on a particular model's capabilities.

ChatGPT identifies an application rather than necessarily specifying the exact underlying model used for every interaction.

Different model configurations may be available through the same interface, and deployed behavior may change over time.

Researchers should therefore identify the underlying model when that information is available.

They should also document the testing period and relevant interaction conditions.

If the interface does not reveal the precise model version, the manuscript should explain that limitation.

The objective is to identify the actual research subject rather than infer technical details from a product name.

What is the difference between a model alias and a snapshot?

A model alias is a name that may refer to a provider-selected deployment or version.

Depending on the service, an alias may be updated to point to a newer implementation.

A snapshot identifier is intended to identify a particular release or version more specifically.

However, researchers should verify the provider's documented semantics rather than assume that every dated identifier is immutable.

For example, an API may expose both a general model identifier and a more specific version designation.

Where available, researchers should preserve the specific identifier used for the request and relevant response metadata.

Even a fixed model snapshot may depend on supporting infrastructure that is not completely controlled by the researcher.

Can researchers determine the exact model version from its answers?

Generally, not reliably.

A model may produce statements about its identity, but those statements should not automatically be treated as authoritative technical metadata.

For example, asking a chatbot which model version it is using may produce an answer that does not correspond to a verifiable deployment identifier.

Researchers should prefer provider documentation, API metadata, authenticated configuration records, or other reliable system information.

Where such evidence is unavailable, the precise version should be reported as unknown.

Researchers should not infer an exact model checkpoint solely from its writing style, apparent capabilities, or self-description.

What if the model version is not publicly disclosed?

The study may still be scientifically valuable, but its reporting should reflect the limitation.

Researchers can document the provider, interface, visible model designation, access method, testing dates, prompts, and generated outputs.

For example, a study may identify its subject as a particular commercial chatbot service accessed during a specified collection period, with no immutable model snapshot exposed.

This is more transparent than inventing an exact version or implying that the deployment remained unchanged.

The research may support findings about the observed service without allowing precise reconstruction of its internal configuration.

Should researchers record model versions when AI is only a research assistant?

The need for detailed version documentation depends on the significance of the AI assistance.

For example, using AI for minor language editing may not require the same methodological detail as using it to classify thousands of research records.

However, applicable journal or institutional disclosure requirements may still apply.

When model outputs materially influence data collection, analysis, interpretation, or other consequential procedures, version information becomes more important.

Researchers should document enough information to explain how the AI-assisted procedure was conducted and evaluated.

The level of detail should be proportionate to the model's role in the research.

Why is version tracking essential when AI itself is the research subject?

When the research question concerns AI performance, the system's identity directly affects the interpretation of the findings.

For example, a study may report that a model answered 82 percent of a particular question set correctly.

Without identifying the tested version or deployment period, readers may not know which system the performance estimate describes.

Subsequent model revisions could produce different results.

The methodological distinction described in studying AI as the research subject therefore makes system identification a central reporting requirement.

Researchers should avoid broad claims about an entire model family when only one identifiable deployment was evaluated.

Does recording the exact model version guarantee reproducibility?

No. Exact version identification addresses only one component of reproducibility.

Model outputs may also depend on prompts, conversation history, generation parameters, tool access, computational implementation, and other conditions.

Even when a particular model snapshot is available, repeated requests may produce different responses.

Researchers should therefore preserve the broader testing procedure.

The distinction between identifying the system and reproducing its behavior is important.

Model version documentation improves provenance, but reproducibility requires attention to the entire computational workflow.

Should researchers record temperature and other generation settings?

Yes, when those settings are available and relevant to the research.

Generation parameters may influence response variability, output length, or other observable characteristics.

For example, a study evaluating response stability should document whether temperature or other sampling controls were configured.

However, not all interfaces expose these parameters.

Researchers should report which settings were explicitly controlled and which remained provider-managed.

They should not assume that unavailable parameters had particular values unless supported by documentation.

Does temperature zero make an exact model version deterministic?

Not necessarily.

Temperature settings may influence sampling behavior, but they do not universally eliminate all sources of output variability.

Computational implementations, service infrastructure, and other system components may affect results.

Some models or interfaces may not provide a temperature parameter at all.

Researchers should therefore evaluate actual response stability when reproducibility matters.

Reporting an exact model version and temperature setting does not establish that every repeated request will produce identical text.

Should researchers record the access method?

Yes. Accessing a model through a web interface may not be equivalent to accessing it through an API or another deployment arrangement.

Interfaces may differ in available tools, hidden instructions, context handling, retrieval capabilities, and output processing.

For example, a chatbot application with web retrieval enabled may produce different answers from a direct model API request without retrieval.

Researchers should identify the access method and relevant features used during testing.

This helps distinguish the behavior of the underlying model from that of the broader application.

The research question should determine whether the object of evaluation is the model, the deployed service, or the complete user-facing system.

What if a provider updates the model without changing its name?

Researchers may be unable to establish that the underlying system remained identical across testing periods.

They should therefore preserve temporal information and relevant observations.

If behavior changes, researchers can investigate whether the difference reflects ordinary variability or a more persistent shift.

However, behavioral change alone does not prove that a particular technical update occurred.

The challenges discussed in studying changing commercial AI models are especially relevant when model names function as changing service identifiers.

Researchers should report observable differences without asserting undocumented technical causes.

How should model versions be handled in longitudinal studies?

Longitudinal studies should record the model or deployment information available at each testing period.

For example, researchers may evaluate the same AI service monthly using a stable benchmark.

If the provider exposes version changes, those changes can be documented alongside the observations.

If exact versions are unavailable, the study should still preserve testing timestamps and visible identifiers.

Researchers may examine whether performance changes across periods, but they should distinguish temporal association from causal attribution to a model update.

The analytical design should account for the possibility that the system being observed is not technically identical throughout the study.

Can model version information help explain conflicting research findings?

Potentially. Two studies may evaluate different versions of a model while using similar tasks.

For example, one study may report frequent citation hallucinations while another reports substantially fewer errors.

Version differences could contribute to the discrepancy.

However, other factors may also matter, including prompt wording, evaluation criteria, task selection, retrieval access, and testing conditions.

Researchers should therefore examine the complete methodology before attributing disagreement to model updates.

Version information helps identify possible explanations but does not independently establish causality.

Should researchers archive model files or weights?

When using locally available models and where licensing permits, preserving model files or verifiable identifiers may improve reproducibility.

For example, researchers may record a model repository revision, checkpoint identifier, or cryptographic hash.

However, model weights alone may not reproduce the complete system.

Tokenizers, inference libraries, quantization settings, hardware, and other dependencies can influence behavior.

Researchers should therefore document the relevant computational environment.

For commercial hosted systems, archiving model weights may not be possible because the provider does not make them available.

Can a model version identifier be included in a research dataset?

Yes. Version information may be stored as metadata associated with each observation or testing session.

For example, a dataset containing AI-generated answers may include fields identifying the model, testing timestamp, prompt identifier, and generation run.

This can be particularly useful when comparing several models or testing across multiple periods.

However, researchers should distinguish actual provider-supplied identifiers from labels assigned internally for project management.

A researcher-created run identifier does not establish the underlying model's exact technical version.

Metadata definitions should be documented in a data dictionary or accompanying research record.

What is a practical model provenance record?

A model provenance record documents the information needed to identify the system and the conditions under which observations were generated.

For example, a research team might preserve the following fields:

The values are illustrative rather than a record of an actual model evaluation.

Researchers may adapt the structure to their study, provided that the resulting metadata accurately represent what was known and controlled.

Should researchers include version information in the manuscript?

Generally, yes, when model identity materially affects the methods or findings.

The methods section should identify the model and relevant version information at a level appropriate to the study.

More detailed metadata may be placed in supplementary materials or a research repository.

For example, a manuscript may identify the model provider, specific API identifier, testing dates, and relevant configuration.

If the exact snapshot was unavailable, that limitation should be stated.

The objective is to make the evaluated system identifiable without overwhelming the main text with incidental technical details.

How should researchers cite an AI model?

Researchers should follow the applicable journal or disciplinary citation requirements.

A citation may identify the provider, model or product, version where available, and relevant access information.

However, citing the provider's website does not replace methodological documentation.

For example, a general citation to a chatbot application may not identify the exact model used during data collection.

Researchers should distinguish bibliographic attribution from the technical provenance required for reproducibility.

Where appropriate, model documentation and archived testing records can supplement the citation.

What if the exact version cannot be recovered after data collection?

Researchers should avoid inventing a version identifier or inferring one from unsupported assumptions.

They may examine existing research logs, API records, provider documentation, or other legitimate sources of provenance.

If the exact version remains unknown, the manuscript should report the available information and acknowledge the limitation.

Preserved prompts and outputs may still support analysis of the observed service behavior.

However, claims requiring identification of a particular immutable model snapshot may need to be narrowed.

The absence of perfect provenance should be disclosed rather than concealed.

Does model version documentation matter for research integrity?

Yes, particularly when model identity is consequential to the findings.

Accurate documentation helps readers understand what system was evaluated and prevents unsupported claims about reproducibility.

However, failing to record a version does not automatically constitute research misconduct.

The significance depends on the research design, reporting requirements, and whether the omission materially misrepresents the work.

Researchers should distinguish incomplete documentation from deliberate falsification.

The appropriate response to missing version information is transparent reporting and, where necessary, qualification of the conclusions.

Recording Model Identity in an AI Accuracy Study

The resulting provenance record supports independent examination of the evaluation procedure.

However, it does not guarantee that future requests to the commercial service will produce identical responses.

Accurate model identification strengthens reproducibility while preserving an appropriate account of the remaining uncertainty.

Common Misconceptions About Recording AI Model Versions

Record What the Provider Actually Exposes, Not What You Assume the System Must Be

Researchers should establish model-identification procedures before formal data collection.

The objective is to preserve the most precise available information while avoiding unsupported claims about inaccessible technical details.

Researchers should also distinguish the underlying model from the broader service through which it was accessed.

Precise documentation improves the interpretability of findings even when exact technical reconstruction remains impossible.

Before Recording AI Model Information in a Research Study

Frequently Asked Questions About Recording AI Model Versions

Precise Model Identification Strengthens Research Provenance Without Guaranteeing Identical Results

How to Cite This Guide

This guide is intended to be read, shared, and used in research, teaching, and academic work. If you draw on its ideas, explanations, or other content, please acknowledge the source by citing the guide. Doing so gives appropriate credit and helps your readers locate the original resource.