03 · What You Need to Know
How to Maintain Research Integrity When AI Model Versions Cannot Be Controlled
Why Might an AI Provider Not Offer Model Version Locking?
Commercial AI services are often designed as continuously maintained products rather than permanently fixed research instruments.
Providers may update models to improve capabilities, address security concerns, modify safety mechanisms, or maintain their technical infrastructure.
Some services expose specific model releases through an API. Others provide general model aliases or consumer-facing interfaces that do not allow users to identify or preserve the exact underlying implementation.
Even when a provider offers version-specific identifiers, those versions may eventually be deprecated.
Researchers should distinguish between a service that provides no version controls, one that provides limited controls, and one that offers fixed releases for a defined support period.
The precise arrangements should be verified through current provider documentation rather than inferred from the product name.
Why Is the Absence of Version Locking a Scientific Concern?
Research depends on knowing which procedures produced the observations being analyzed.
When an AI model contributes directly to classification, scoring, information extraction, or other measurements, changes in the system may alter the resulting variables.
Suppose a study uses an AI assistant to identify whether research abstracts describe randomized experiments. The provider changes the underlying model midway through processing.
If the updated system applies different classification criteria, the frequency of identified experiments may change even when the underlying documents have similar characteristics.
Without adequate records, researchers may be unable to determine whether the difference reflects the documents or the AI system.
This is why locking a model version when possible is often preferable for studies requiring stable measurement.
Does the Lack of Version Locking Make Research Irreproducible?
It can limit exact computational reproducibility, but the implications require careful distinction.
Exact computational reproduction involves rerunning substantially the same computational procedure under equivalent conditions. If the original model becomes unavailable, another researcher may be unable to reproduce the exact outputs.
However, the original study may still be scientifically interpretable if the researchers preserved the inputs, outputs, evaluation procedures, model information available at the time, and evidence supporting the validity of their measurements.
Independent replication may also be possible using a different AI system or an alternative measurement procedure, although that would not constitute exact re-execution of the original computational process.
Exact computational reproducibility
The ability to rerun the original computational procedure under sufficiently equivalent conditions.
Scientific interpretability
The ability to understand what was done, how evidence was produced, and what conclusions the available observations can support.
Researchers should not claim exact reproducibility when the underlying implementation cannot be identified or restored.
At the same time, limited technical access does not automatically make every finding scientifically meaningless.
What Information Should Researchers Record if the Model Version Is Unknown?
When exact version information is unavailable, researchers should preserve the most specific reliable information that can be obtained.
Useful records include the product name, provider, access method, processing dates, visible model identifiers, prompts, configuration settings, input characteristics, and relevant output files.
For API-based workflows, request identifiers, response metadata, and processing logs may provide additional traceability.
For browser-based interfaces, researchers may have fewer technical records available. They should document the interface used and any relevant settings or features.
Where ethically and legally permissible, preserving the actual input-output pairs is particularly valuable.
Researchers should also record whether retrieval tools, file-processing features, or external services were enabled.
These details do not establish the exact model version, but they help characterize the system as it was encountered during the study.
How Can Researchers Detect Changes Without Knowing the Underlying Model Version?
One practical approach is to maintain a fixed evaluation dataset containing representative inputs and independently established reference answers.
The research team periodically submits these inputs to the AI service using a documented procedure and compares the resulting outputs with earlier observations.
Such a dataset can function as a set of monitoring or sentinel cases.
It may include common research tasks, difficult examples, ambiguous cases, and inputs associated with consequential errors.
However, changes in outputs do not necessarily prove that the underlying model changed. Probabilistic generation, retrieval differences, or other operational factors may also affect responses.
Repeated evaluation and appropriate reference standards can help distinguish ordinary variability from systematic behavioral changes.
The objective is to detect changes that matter for the research task, not to infer undisclosed technical modifications from every different answer.
What Is a Sentinel Evaluation Set?
A sentinel evaluation set is a fixed collection of test cases used repeatedly to monitor whether a system's behavior or performance has changed.
For example, a research team studying automated abstract screening might preserve 100 abstracts with independently adjudicated eligibility decisions.
At predefined intervals, the team evaluates the AI system on those same cases and examines changes in classification accuracy, false exclusions, and other relevant outcomes.
The set should be representative of the intended task and include cases that expose important failure modes.
Its limitations must also be recognized. A small fixed set may fail to detect changes affecting unfamiliar inputs, and repeated optimization against the same cases can reduce their value as independent evaluation material.
Researchers may therefore combine a stable sentinel set with additional independent evaluation samples when resources permit.
How Frequently Should Researchers Test for Changes?
There is no universal monitoring interval.
The appropriate frequency depends on the study duration, processing volume, consequences of errors, and likelihood of meaningful system changes.
A short project processing all documents within a controlled period may require fewer monitoring points than a longitudinal study operating over several months.
For consequential applications, monitoring may need to occur before major processing batches, after documented provider updates, or whenever unexpected behavior is observed.
Researchers should establish the monitoring schedule before data collection when feasible.
Testing should be frequent enough to detect changes that could affect the study, while avoiding unnecessary evaluations that provide little additional information.
Can Researchers Process All Data in a Short Time Window?
Batch processing can reduce exposure to changes occurring over an extended period.
For example, a team might collect research documents over several months but process them through the AI system within a short, documented period.
This can be useful when AI processing occurs after raw data collection and does not influence participant interactions.
However, processing everything within one day does not guarantee that the system remained unchanged throughout that period.
Providers may implement updates without advance notice, and output variability can occur independently of version changes.
Researchers should therefore record processing timestamps and conduct appropriate checks even when using a short processing window.
Should Researchers Separate Raw Data Collection From AI Processing?
Where the research design permits, separating these activities can substantially improve flexibility.
Raw observations can be collected and preserved independently of the AI system. The AI model can then be applied during a controlled analytical phase.
If the model changes unexpectedly, researchers may be able to reprocess the original observations using a different validated system.
This arrangement is particularly useful for document classification, information extraction, and retrospective coding.
It is less feasible when AI outputs influence real-time participant interactions, adaptive interventions, or subsequent data collection.
In those cases, the model's behavior forms part of the data-generating process itself.
What if the AI Provider Changes the System During Data Collection?
Researchers should document the suspected or confirmed change and determine whether it could affect the study's measurements or decisions.
Where possible, compare outputs from before and after the change using common inputs and reference standards.
If a material difference is detected, the team may need to suspend affected processing, revise procedures, or analyze observations separately.
Not every behavioral difference requires restarting the study.
The appropriate response follows the principles of managing AI changes during an ongoing research project.
Researchers should preserve the distinction between documented provider updates and changes inferred only from observed outputs.
Can Researchers Combine Outputs Collected at Different Times?
Potentially, but time-separated outputs should not automatically be treated as equivalent.
If the AI system contributes directly to research variables, researchers should assess whether its measurement behavior remained sufficiently stable across processing periods.
A fixed sentinel dataset may provide evidence of continuity, but it cannot establish equivalence across every possible input.
When systematic differences appear, researchers may need to investigate whether the outputs can be harmonized, reprocessed, or analyzed separately.
The decision should be based on evidence supporting the comparability of AI-generated measurements.
Merely using the same prompt or output format does not establish that comparability.
Should Researchers Use a Different AI Provider Instead?
Changing providers may be appropriate when the absence of version control creates unacceptable methodological risks.
However, another provider should not be assumed more suitable simply because it offers a fixed model identifier.
Researchers should evaluate the alternative system's task performance, privacy arrangements, availability, costs, and technical requirements.
A locally deployed or institutionally hosted model may offer greater control over version identity, but it can introduce infrastructure and maintenance responsibilities.
For some projects, a validated non-AI method may be more defensible than relying on a continuously changing AI service.
The decision should reflect the scientific requirements of the study rather than a preference for a particular deployment model.
Can Researchers Use Multiple AI Models to Reduce Dependence on One Provider?
Multiple models can provide useful comparative evidence and may reduce operational dependence on one service.
However, agreement among models does not guarantee correctness, particularly when they share training sources, similar architectures, or common failure modes.
Using several models also complicates the research procedure because each system may have different capabilities, configurations, and update schedules.
Researchers should not treat majority agreement as a substitute for independent reference standards.
Multiple-model evaluation is most useful when it addresses a clearly defined methodological question rather than serving as an informal attempt to compensate for unavailable version control.
When Is Research Without Version Locking Too Risky?
The absence of fixed-version access becomes especially problematic when the study requires stable measurements over long periods, when errors could materially harm participants, or when the original system must be available for exact computational reproduction.
It may also be unsuitable when the provider does not supply enough information to identify consequential changes and researchers cannot establish independent safeguards.
For example, a longitudinal study that relies on AI-generated primary outcome scores may face substantial interpretive difficulties if the scoring system changes unpredictably and original inputs cannot be reprocessed.
In such circumstances, researchers should consider a different system, a redesigned measurement procedure, or a non-AI alternative.
Watch Out
Systematic logging and monitoring reduce uncertainty but do not recreate technical control that the provider does not offer. When the research requires an immutable instrument or exact re-execution, documentation alone cannot make a changing service equivalent to a fixed model.
How Should Researchers Report Studies Conducted Without Version Locking?
The methods section should identify the provider, product, access method, processing period, and all available system identifiers.
Researchers should explain that fixed-version access was unavailable and describe the procedures used to monitor system behavior.
Relevant details may include sentinel datasets, evaluation intervals, reference standards, processing batches, archived outputs, and any observed changes.
If the system changed or comparability remained uncertain, those limitations should be incorporated into the analysis and interpretation.
Researchers should avoid claiming exact reproducibility when the original implementation cannot be reconstructed.
Transparent reporting helps readers determine whether the study remains interpretable after the underlying AI system changes.
