What Should Researchers Do When an AI Model Changes During a Study?

What Happens When the AI System Used in Your Study Changes Before the Research Is Finished?

You are halfway through collecting data for a study that uses an AI model to classify research documents, evaluate student responses, or extract information from interview transcripts. The provider announces a model update. Alternatively, you notice that the system has begun producing different outputs despite using the same prompts.

Should you continue collecting data? Must you repeat earlier analyses? Can results generated before and after the update still be combined?

These questions matter because the AI system may function as part of the research instrument or measurement procedure. A change in its behavior could introduce differences that are unrelated to the phenomenon being studied.

Document the Change, Assess Its Impact, and Protect Comparability

How AI Model Changes Can Affect an Ongoing Research Study

Why Is an AI Model Update a Methodological Concern?

Research methods depend partly on the consistency of the procedures used to produce and interpret observations. When an AI model contributes to those procedures, changes to the model may affect the resulting data.

Consider a study in which an AI system classifies participants' written responses into categories. If the classification criteria applied by the system change halfway through data collection, apparent differences between early and late participants may reflect the altered classifier rather than genuine differences in participant responses.

The problem resembles a change in a measurement instrument, coding protocol, or laboratory procedure during an investigation.

However, not every AI update is necessarily a substantive methodological change. Researchers must determine whether the modification affects the properties relevant to their study.

What Types of AI Changes Should Researchers Investigate?

Model updates are not limited to changes in model weights. The complete AI workflow may change through modifications to surrounding software or services.

Some changes may be documented in official release notes. Others may not be fully disclosed, particularly when researchers use managed commercial services.

When the provider does not identify the exact change, researchers should distinguish observed behavioral differences from assumptions about their technical cause.

Does the AI Model's Role Determine How Serious an Update Is?

Yes. The methodological consequences depend substantially on how the AI system contributes to the research.

If the system assists with brainstorming manuscript titles, a change in its outputs may have little effect on the scientific evidence.

If it extracts the primary outcome variable, determines participant eligibility, or assigns scores used in statistical analysis, the same kind of update could have substantial consequences.

The more directly an AI output influences the study's primary findings or participant welfare, the stronger the justification needed for continuing without additional assessment.

How Can Researchers Find Out Whether a Model Has Changed?

Start with the information provided by the system or its developer.

For API-based research, inspect model identifiers, release documentation, deprecation notices, and relevant service records. Some services offer fixed model versions, while others expose aliases that can change their underlying implementation.

For locally deployed systems, researchers may be able to record model file hashes, software versions, configuration files, and dependency information.

For consumer-facing interfaces, exact model identification may be more difficult. Product names and visible interface labels do not necessarily identify every component responsible for the output.

Researchers should therefore document what is known, what is inferred, and what remains unavailable.

A change in outputs alone does not prove that a model update occurred. Stochastic generation, changed conversation context, altered retrieval results, or differences in inputs may also explain the observation.

Should Researchers Pause Data Collection Immediately?

Not automatically. The decision should be based on the likely consequences of continuing before the change has been assessed.

A temporary pause may be appropriate when the AI system generates primary measurements, influences consequential participant decisions, or produces outputs that cannot later be reconstructed.

Continuing may be defensible when the system's role is peripheral, the change is demonstrably irrelevant to the study's outcomes, or adequate safeguards prevent affected outputs from influencing conclusions before verification.

Where participant safety or ethical commitments may be affected, researchers should follow the applicable approved protocol and consult the relevant oversight body.

The more detailed decision about whether data collection should be paused after an AI update depends on the nature and urgency of the risk.

How Should Researchers Test Whether the Update Changed Performance?

A defensible approach is to compare the original and updated systems on a common evaluation dataset using the same task definitions and reference standards.

Where the earlier model remains accessible, researchers can run both versions on a fixed set of representative inputs. The evaluation should examine not only aggregate performance but also changes in individual outputs and consequential error types.

For classification tasks, relevant measures may include agreement, sensitivity, specificity, and category-specific error rates. For scoring systems, researchers may need to examine score differences, systematic bias, and measurement agreement.

For generative AI tasks, expert evaluation may be required to assess factual accuracy, extraction fidelity, or adherence to a predefined coding rubric.

Repeated runs can help distinguish systematic changes from ordinary response variability, although repeated outputs from one model are not necessarily statistically independent.

The evaluation should be designed to address whether the update materially changed performance, rather than merely whether any output differs.

What if the Original Model Is No Longer Available?

Loss of access complicates direct comparison but does not eliminate all possibilities for assessment.

Researchers may have retained earlier inputs and outputs, evaluation records, reference judgments, or archived system configurations. These materials can support comparisons with the updated system.

However, previously generated outputs may not provide the same evidence as rerunning both versions under identical conditions. Differences in sampling, input selection, and workflow settings can complicate interpretation.

If no adequate baseline exists, researchers should avoid claiming that performance remained unchanged.

They may instead evaluate the updated system against an independent reference standard and acknowledge the limits of comparison with the earlier version.

Future studies can reduce this vulnerability through model version locking where available and systematic preservation of evaluation records.

Can Outputs From Before and After an Update Be Combined?

Potentially, but combining them requires a defensible rationale.

If the AI system generates a research variable, the team should investigate whether that variable has the same meaning and measurement properties across versions.

A model that assigns systematically higher scores after an update may introduce an artificial temporal difference. A classifier that changes its treatment of ambiguous cases may alter category frequencies.

Researchers may need to reprocess earlier data with a single validated version, retain version-specific datasets, include version-related terms in appropriate analyses, or conduct sensitivity analyses.

These options are not interchangeable, and statistical adjustment cannot automatically repair an inadequately understood measurement change.

The methodological issue is whether outputs from different model versions can be combined without compromising interpretation.

When Does a Model Update Require a Research Protocol Amendment?

Requirements vary across institutions, funders, ethics committees, and regulatory settings.

A protocol amendment may be necessary when the change affects approved procedures, participant-facing activities, risk management, data processing, or substantive elements of the study design.

Some minor operational adjustments may fall within an existing approved protocol. Researchers should not assume that all AI updates automatically require formal amendment, nor that technical changes are exempt from review.

When uncertain, consult the institution's research governance procedures or the relevant ethics review body before implementing a consequential change.

How Should Researchers Document a Model Change?

A useful change record should identify the original and updated systems, the date the change was detected or implemented, the affected research procedures, and the evidence used to assess its consequences.

It should also record the decision taken, who authorized it, and how the study's analysis or reporting was modified.

Where available, preserve prompts, configurations, evaluation datasets, output samples, code, and version identifiers, subject to privacy and licensing restrictions.

These records help distinguish a planned methodological adaptation from an undocumented change that could compromise interpretation.

Documentation also makes it possible for readers to understand how the study remains interpretable after the AI system changes.

Managing an AI Model Update During Educational Research Data Collection

The central concern is not merely that 25 of the 60 scores changed. The team must determine whether the differences reflect measurement error, improved rubric alignment, systematic scoring shifts, or ordinary variability.

If the AI-derived scores influence the study's primary conclusions, the team should also examine whether alternative defensible scoring procedures materially change the results.

Common Mistakes When AI Models Change During Research

How to Respond to an AI Update Without Compromising Your Study

Researchers should establish a change-management procedure before relying on an AI system for consequential data collection or analysis.

At minimum, the procedure should define how updates will be detected, what evidence will trigger reassessment, who will decide whether research activities can continue, and how changes will be documented.

When possible, avoid making a commercial model's continued availability an unexamined assumption of the research design. Investigate how commercial AI dependencies may affect research continuity before beginning long-term data collection.

The appropriate response should preserve scientific interpretability rather than merely minimize disruption to the research schedule.

What to Check When an AI Model Changes During Your Study

Frequently Asked Questions About AI Model Updates During Research

An AI Model Update Is a Research Change That Must Be Evaluated, Not Automatically Feared

How to Cite This Guide

This guide is intended to be read, shared, and used in research, teaching, and academic work. If you draw on its ideas, explanations, or other content, please acknowledge the source by citing the guide. Doing so gives appropriate credit and helps your readers locate the original resource.