03 · What You Need to Know
How Changing AI Models Affect Research Reproducibility
What Does Reproducibility Mean in AI-Assisted Research?
Reproducibility is often used broadly to describe whether research findings can be obtained again. However, scientific methodology distinguishes several related concepts.
The National Academies of Sciences, Engineering, and Medicine define reproducibility in computational research in terms of obtaining consistent computational results using the same input data, computational steps, methods, code, and conditions of analysis.
Replicability concerns obtaining consistent results across studies intended to address the same scientific question using new data.
AI-assisted research adds complications because the computational system may not remain available in the same form.
Researchers should therefore specify what they intend to reproduce: an exact generated response, a classification procedure, an aggregate analytical result, or a broader scientific conclusion.
Why Can the Same Prompt Produce Different Outputs?
Generative AI systems may produce different responses to identical prompts because generation can involve probabilistic sampling and other sources of computational variability.
Output differences may also arise from model updates, changes in system instructions, retrieval results, tool configurations, or infrastructure.
Some interfaces expose parameters such as temperature or random seeds, while others provide limited control over generation settings.
Even when a seed or deterministic-looking configuration is available, identical output is not necessarily guaranteed across different hardware, software versions, or deployment conditions.
Researchers should not assume that saving a prompt alone preserves the complete computational procedure.
What Happens When a Provider Updates the Model?
Model updates may alter how a system interprets instructions, applies categories, handles uncertainty, or generates responses.
Some changes may improve performance, while others may affect particular tasks in ways that are not immediately apparent.
For example, a revised model might interpret ambiguous student feedback differently, even when the researcher uses an unchanged classification prompt.
Such changes are especially important when outputs from different periods are compared as though they were generated by an identical measurement instrument.
A provider's product name may remain unchanged while underlying components or settings are modified. Researchers should therefore examine whether the service offers fixed model snapshots, version identifiers, or other controls relevant to their study.
Why Model Changes Matter More for Some Research Tasks
The consequences of model variability depend on the AI contribution.
| AI task |
Effect of model changes |
Reproducibility concern |
| Brainstorming research questions |
Different suggestions may be generated |
Usually limited if ideas are subsequently evaluated independently |
| Language editing |
Different wording may be suggested |
Often manageable if scientific meaning is preserved |
| Literature screening |
Different records may be included or excluded |
Potentially consequential for the completeness of the evidence base |
| Qualitative classification |
Categories or assignments may change |
May affect reported themes and comparisons |
| Numerical scoring |
Scores may differ between model versions |
May undermine measurement comparability |
| AI-generated analytical code |
Different implementations may be produced |
Can affect calculations if code is regenerated rather than preserved |
| Longitudinal research measurement |
Outputs may change independently of the underlying phenomenon |
May confound genuine temporal change with measurement change |
Model instability is not automatically unacceptable. Its importance depends on whether changes can alter consequential evidence or decisions.
Why Longitudinal Studies Face Particular Difficulties
Longitudinal research examines change over time. Comparability of measurement is therefore essential to interpreting observed differences.
Suppose an AI classifier assigns engagement categories to student reflections collected in successive academic years.
If the underlying classifier changes, differences in category frequencies may reflect changes in the students, changes in the AI system, or both.
This creates a measurement comparability problem.
Researchers may need to preserve a fixed model, reprocess all periods using a common validated procedure, or establish comparability between model versions through appropriate evaluation.
Without such controls, attributing observed differences entirely to the phenomenon under study may be unjustified.
Does Using a Fixed Model Version Solve the Problem?
A fixed model version can reduce uncertainty caused by provider updates, but it does not automatically guarantee identical outputs.
Generation settings, computational environment, stochastic processes, external tools, and data preprocessing may still influence results.
Researchers should therefore document the model version alongside other relevant components of the workflow.
Where possible, preserving the original code, inputs, outputs, and evaluation procedures provides stronger evidence than recording the model name alone.
Fixed versions are especially useful when model behavior forms part of the research measurement process, but they remain one component of a broader reproducibility strategy.
Can Random Seeds Make Generative AI Fully Reproducible?
A random seed may help control some stochastic operations by initializing a pseudorandom process.
However, a seed does not necessarily control every source of variability in a deployed AI system.
Differences in model weights, numerical implementation, hardware, software libraries, inference infrastructure, or external retrieval can still affect outputs.
Researchers should consult the specific provider's documentation to determine what reproducibility guarantees, if any, are supported.
A seed should not be described as guaranteeing identical results unless that claim is supported for the relevant system and conditions.
When Is Statistical Reproducibility More Important Than Identical Text?
Some research applications do not require the same wording to be generated on every run.
For example, a study might use repeated model runs to estimate the distribution of responses under a specified prompt and configuration.
In such a design, variation is part of the phenomenon being examined rather than necessarily an error.
Researchers should specify the sampling procedure, number of runs, configuration, outcome measures, and uncertainty associated with the estimates.
The findings may be reproducible at the level of a distribution or aggregate pattern even when individual generated responses differ.
However, this approach is inappropriate when the study requires exact classification of individual observations and the variability cannot be adequately controlled.
How Can Researchers Test Whether Model Changes Matter?
When model versions differ, researchers can evaluate their effects using a common reference dataset and a prespecified comparison procedure.
For a classification task, this may involve applying both versions to the same observations and comparing their outputs against independently established reference labels.
Relevant measures may include agreement, class-specific performance, error patterns, and changes in aggregate results.
Researchers should also examine whether disagreements concentrate in particular subgroups, categories, or types of observations.
High overall agreement may conceal consequential changes in rare but important categories.
Where the comparison reveals material differences, results from the two versions should not automatically be treated as directly interchangeable.
What Is a Bridge Study Between AI Model Versions?
A bridge study, in this context, is an evaluation designed to determine whether outputs from different model versions can be meaningfully compared or combined.
The term describes a possible methodological approach rather than a universally standardized requirement for generative AI research.
Researchers may select a reference set representing the intended research population, apply both model versions under documented conditions, and examine differences in relevant outputs.
If performance is sufficiently comparable for the intended purpose, continued use may be defensible with appropriate documentation.
If differences are substantial, the researcher may need to recalibrate the procedure, reprocess earlier data, or analyze periods separately.
Comparability should be established against the scientific requirements of the study, not inferred merely because two versions produce similar-looking responses.
Should Researchers Archive AI Outputs?
Preserving consequential AI outputs can support auditing and later examination, particularly when the original model becomes unavailable.
For example, retaining classification results, associated record identifiers, prompts, and validation decisions allows researchers to inspect the original analysis without requiring the provider to reproduce the same response.
However, archived outputs do not establish that the procedure can be rerun identically. They document what occurred during the original execution.
Archiving must also respect confidentiality, data protection, copyright, and research ethics obligations.
Where outputs contain participant information, the archive should use an approved protected environment rather than an unrestricted public repository.
When Should Researchers Reprocess Earlier Data?
Reprocessing may be appropriate when a model update changes the measurement procedure used across different periods or datasets.
For example, a longitudinal study may apply one fixed, validated classifier to all previously collected responses to improve comparability.
However, reprocessing is not automatically the correct solution. Earlier inputs may be unavailable, data-processing conditions may have changed, or the new model may introduce different biases.
Researchers should consider whether reprocessing preserves the intended construct, whether the model has been validated across all relevant periods, and whether changes in the dataset itself affect comparability.
Where necessary, sensitivity analyses can help establish whether the study's conclusions depend on the model version selected.
When Is a Non-AI Method Preferable?
A deterministic or otherwise stable analytical procedure may be preferable when the task requires consistent outputs under defined conditions.
For example, applying an explicit rule to classify records based on a fixed numerical threshold may be more transparent and reproducible than asking a generative AI system to interpret the same rule repeatedly.
Similarly, established statistical software may be preferable for calculations that do not benefit from generative inference.
This does not mean conventional methods are free from versioning or implementation problems. Software updates, dependencies, and data-processing decisions must also be documented.
The comparison should focus on which procedure best satisfies the research requirements with manageable uncertainty and adequate validation.
When Does Model Instability Make AI Use Scientifically Indefensible?
Model instability becomes a fundamental concern when changes can materially alter the reported findings and the research team cannot identify, measure, or control those changes sufficiently.
For instance, a study claiming that student sentiment improved over time may be difficult to defend if different, undocumented AI models classified responses from different years and no evidence establishes comparability.
The same concern arises when an AI-generated score is treated as a stable measurement despite substantial unexplained variation across repeated runs.
In these circumstances, the researcher should consider a fixed model, a validated alternative procedure, or a revised design that explicitly incorporates model variability.
Watch Out
Do not interpret changes in AI-generated classifications or scores as changes in the research phenomenon until you have considered whether the measurement system itself changed.
