Can Researchers Combine AI Outputs Collected Before and After a Model Update?

Can AI-Generated Research Data From Two Model Versions Be Analyzed Together?

A researcher uses an AI model to classify interview responses, extract variables from scholarly articles, or score student writing. Halfway through the project, the AI provider updates the model. The remaining observations are processed using the newer version.

Both versions produce the same categories or numerical scores. Does that mean their outputs can be combined into one dataset?

Not necessarily. If the versions apply different decision boundaries, interpret instructions differently, or produce different error patterns, pooling their outputs may introduce artificial differences into the analysis.

The methodological question is whether observations generated by different AI versions remain sufficiently comparable for the intended scientific interpretation.

Combining AI Outputs Is Possible When Their Comparability Is Defensible

When AI Outputs From Different Versions Can Support a Common Analysis

Why Is Combining AI Outputs a Measurement Problem?

When AI-generated classifications, scores, or extracted values become research variables, the AI system forms part of the measurement procedure.

For the resulting observations to be meaningfully compared, researchers need confidence that the procedure produces sufficiently consistent interpretations across relevant conditions.

Suppose an AI model scores students' critical reflection on a five-point scale. The original version emphasizes evidence-based reasoning, while the updated version places greater weight on writing fluency. Both produce scores from 1 to 5, but those scores may no longer represent the intended construct in the same way.

Combining them without investigation could introduce measurement differences that are incorrectly interpreted as differences between students or changes over time.

This concern is closely related to treating AI updates as potential research instrument changes.

What Does Comparability Mean for AI-Generated Data?

Comparability means that outputs produced under different measurement conditions can support the same intended interpretation with an acceptable degree of uncertainty.

It does not necessarily require every individual output to be identical.

For example, two validated human raters may occasionally disagree while still producing measurements sufficiently consistent for a particular research purpose. Similarly, two AI versions may produce some different outputs without making every combined analysis invalid.

The acceptability of those differences depends on their magnitude, pattern, and consequences.

A high output-agreement rate can support comparability, but it does not establish validity if both versions make similar errors.

What Types of AI Outputs May Be Combined?

The required evidence depends on the type of output and its role in the analysis.

Pooling numerical measurements is not the same as combining qualitative interpretations. Researchers should use evaluation methods appropriate to the data and the epistemological assumptions of the study.

How Can Researchers Test Comparability Between Model Versions?

A useful approach is to construct a bridge sample: a representative collection of observations processed by both model versions.

Where possible, the sample should cover the range of values, difficult cases, relevant subgroups, and conditions likely to influence the outputs.

The comparison should use the same original inputs, task instructions, and reference standards wherever feasible.

Researchers can then examine whether differences are random, systematic, concentrated in particular categories, or associated with participant or document characteristics.

For consequential research variables, an independent reference standard is particularly valuable. Agreement between model versions alone cannot establish that either version measures the intended construct correctly.

When planning the comparison, researchers should distinguish material changes in model performance from differences that are inconsequential for the intended analysis.

Is High Agreement Enough to Justify Pooling?

Not always. Overall agreement can conceal important patterns.

Consider a classifier that labels most observations as negative. Two versions might agree on 95% of cases while disagreeing disproportionately on rare positive cases that are central to the research question.

Similarly, two scoring systems may have highly correlated outputs while one systematically assigns higher scores.

Researchers should examine the distribution and severity of disagreements, not merely an aggregate agreement percentage.

For continuous measurements, the Bland–Altman approach provides a framework for examining agreement and systematic differences between measurement methods. For categorical data, agreement statistics and confusion matrices may be more appropriate.

The statistical method should match the measurement scale and intended interpretation.

Should Researchers Define an Acceptable Difference Before Comparing Versions?

Where feasible, yes. Defining an acceptable difference before examining the results helps prevent researchers from selecting a convenient threshold after observing the data.

An equivalence margin should reflect the substantive consequences of measurement differences.

For example, a one-point discrepancy on a five-point assessment scale may be consequential if it changes a student's classification into a predefined proficiency category. The same numerical difference may be less consequential in another analytical context.

Researchers should justify the margin using the measurement purpose, existing validation evidence, expert judgment, or relevant methodological standards.

Statistical nonsignificance is not evidence of equivalence. A small study may fail to detect an important difference simply because its estimates are imprecise.

When formal equivalence testing is appropriate, the analysis should assess whether the confidence interval for the relevant difference falls within a justified equivalence margin.

What if the Model Update Occurs at the Same Time as a Change in the Study Population?

This creates a particularly difficult interpretation problem.

Suppose the original model processes responses from the first semester, while the updated model processes responses from the second semester.

If the later scores are higher, the difference could reflect a genuine change in participants, a change in the model, or both.

When model version and study period are completely aligned, their effects may be statistically inseparable without additional information or design features.

A bridge sample processed by both versions can help identify measurement differences. Reprocessing a common collection of original observations may also provide useful evidence.

Simply adding a model-version indicator to a regression does not resolve perfect confounding between version and time.

Can Statistical Adjustment Make Outputs From Different Versions Comparable?

Sometimes, but only under defensible assumptions.

If the relationship between versions can be estimated reliably using overlapping observations, researchers may be able to develop a calibration or harmonization procedure.

However, a simple numerical correction may be inadequate when the update changes category definitions, construct interpretation, or subgroup-specific error patterns.

Statistical adjustment also introduces uncertainty that should be reflected in subsequent analyses.

Researchers should avoid treating harmonization as a purely computational problem when the underlying measurement meaning may have changed.

Is Reprocessing All Data With One Model Version Better Than Pooling Mixed Versions?

Reprocessing can be a strong option when original inputs remain available and a single validated version can be applied consistently.

It may eliminate version-related differences in the measurement procedure, although ordinary model variability and other sources of error remain.

However, reprocessing is not always possible or desirable.

If AI outputs influenced participant interactions, adaptive interventions, or real-time decisions, later reprocessing cannot recreate the original experience.

Likewise, a newer model should not automatically replace an earlier one merely because it is more recent. Its suitability for the research task must be established.

When an update occurs during data collection, researchers should consider whether affected processing should be temporarily paused while comparability is evaluated.

When Should Researchers Analyze Model Versions Separately?

Separate analysis may be appropriate when the versions measure meaningfully different quantities, when equivalence cannot be demonstrated, or when the research question explicitly concerns differences between versions.

Version-specific analyses can preserve transparency and avoid presenting heterogeneous measurements as interchangeable.

However, separating data may reduce statistical precision and complicate longitudinal interpretation.

Researchers should also examine whether each version is confounded with participant characteristics, collection periods, or other study conditions.

Separate reporting does not automatically solve those design limitations, but it can make them visible.

How Should Researchers Conduct Sensitivity Analyses?

Sensitivity analyses examine whether substantive conclusions change under alternative defensible analytical decisions.

For mixed-version AI data, researchers might compare results obtained from the pooled dataset, version-specific subsets, or a consistently reprocessed dataset.

They may also investigate the effect of excluding uncertain observations or applying a justified calibration procedure.

These analyses should not be used to select whichever result most strongly supports the hypothesis.

Instead, researchers should report how conclusions vary across plausible approaches and explain which assumptions each analysis requires.

If results change substantially depending on how model versions are handled, that instability is itself important methodological information.

What Should Researchers Report When Combining AI Outputs?

The methods section should identify the model versions, processing dates, relevant configurations, and number of observations generated under each version.

Researchers should describe the evidence supporting comparability, including the bridge sample, reference standards, agreement measures, and any predefined acceptance criteria.

If outputs were calibrated, reprocessed, or analyzed separately, those procedures should be explained.

Where appropriate, report sensitivity analyses and limitations related to unavailable versions, incomplete logs, or uncertainty about deployment changes.

These details help preserve the interpretability of the study even when AI performance evidence becomes less applicable to newer systems.

Deciding Whether to Pool Student Reflection Scores From Two AI Models

The hypothetical results do not establish that Model B is less accurate. They indicate that the two versions may not be interchangeable without further evaluation.

Pooling the original scores without investigating the systematic difference could produce an artificial increase in measured critical reflection over time.

Reprocessing with a single validated version may reduce this particular source of inconsistency, but the resulting scores still require appropriate validity evidence.

Common Mistakes When Combining AI-Generated Data Across Versions

How to Choose Between Pooling, Reprocessing, and Separate Analysis

The appropriate decision depends on the measurement role of the AI system and the strength of the available comparability evidence.

Begin by determining whether model version could change the substantive meaning or statistical properties of the research variable.

For long-running studies, establish a version-tracking procedure before data collection begins. Preserve raw inputs and reference judgments where permissible, and consider using a fixed model version when measurement continuity is important.

The central objective is not to make all outputs numerically identical. It is to ensure that combining them does not create misleading evidence about the phenomenon under investigation.

Before Combining AI Outputs Across Model Versions, Check These Points

Frequently Asked Questions About Combining AI Outputs From Different Versions

Pooling AI Outputs Requires Evidence of Comparable Measurement

How to Cite This Guide

This guide is intended to be read, shared, and used in research, teaching, and academic work. If you draw on its ideas, explanations, or other content, please acknowledge the source by citing the guide. Doing so gives appropriate credit and helps your readers locate the original resource.