03 · What You Need to Know
Designing a Defensible Evaluation of AI Model Performance Changes
What Does a Material Change in AI Performance Mean?
A material performance change is a difference large enough to affect the suitability, interpretation, or consequences of using an AI system for a particular purpose.
Materiality is contextual. A small decline in sensitivity may be consequential when an AI classifier screens for rare but important safety risks. A similar numerical difference may have little practical importance in a low-consequence document-organizing task.
Researchers should distinguish three questions: whether the model's outputs changed, whether measured performance changed, and whether the difference matters for the research.
Behavioral change
The updated system produces different outputs or response patterns.
Performance change
The updated system differs in accuracy, agreement with a reference standard, calibration, or another defined performance measure.
Material change
The observed difference is sufficiently consequential for the system's intended research use to influence a methodological or operational decision.
These distinctions prevent researchers from equating every changed answer with a deterioration in scientific reliability.
Why Is a Paired Evaluation Usually Preferable?
When both model versions can process the same observations, a paired design allows researchers to compare their performance case by case.
This reduces the influence of differences in sample composition that would arise if each model were tested on an unrelated collection of inputs.
For example, evaluating Model A on straightforward abstracts and Model B on more ambiguous abstracts would not provide a fair comparison of their classification capabilities.
Instead, both models should receive the same abstracts under appropriately controlled conditions.
Paired testing also allows researchers to identify which cases improved, which deteriorated, and whether errors are concentrated in particular categories.
The design should preserve the pairing during statistical analysis. Treating paired results as independent observations can produce inappropriate uncertainty estimates.
What Should Researchers Keep Constant During the Comparison?
To attribute differences as clearly as possible, researchers should control evaluation conditions that are not intended to vary.
These may include the input dataset, task instructions, output schema, preprocessing procedures, retrieval settings, and reference standards.
However, identical settings are not always technically possible. Different model versions may support different interfaces, parameter ranges, or tool configurations.
When exact matching is impossible, researchers should document the differences and consider whether they affect interpretation.
| Evaluation Element |
Preferred Approach |
Reason |
| Input cases |
Use the same representative observations |
Controls differences in case difficulty and composition |
| Task definition |
Apply the same intended classification or scoring criteria |
Maintains a common performance target |
| Reference standard |
Use consistent, independently established judgments |
Allows comparable assessment of correctness |
| Prompts |
Keep instructions comparable and document unavoidable differences |
Reduces prompt-related confounding |
| Inference configuration |
Control supported parameters where appropriate |
Limits avoidable output variability |
| Evaluation metric |
Apply the same metric definitions and scoring rules |
Supports meaningful numerical comparison |
| Evaluation period |
Test versions under similar conditions when feasible |
Reduces temporal differences in surrounding systems |
Researchers should specify whether they are comparing the underlying models or the complete deployed systems. A comparison of two commercial products may include differences in retrieval, preprocessing, and safety controls that cannot be attributed solely to model weights.
How Should the Evaluation Dataset Be Selected?
The dataset should represent the research tasks and conditions for which the performance decision will be made.
Random sampling from the intended-use population can provide a useful foundation, but researchers may also need to include rare, ambiguous, or consequential cases that would otherwise be underrepresented.
For example, an AI system used to identify eligible studies in a systematic review should be evaluated on difficult inclusion decisions and uncommon study designs, not only straightforward abstracts.
Researchers should distinguish a representative performance sample from a deliberately enriched challenge set. Both can be useful, but results from an enriched set should not automatically be interpreted as population-level performance estimates.
Evaluation cases should not be selected solely because they produced surprising outputs after the update. Doing so can exaggerate apparent changes and introduce selection bias.
What Reference Standard Should Be Used?
A reference standard defines the answer or judgment against which AI outputs are evaluated.
Its construction depends on the task. Document extraction may use verified source information. Classification may require independently adjudicated labels. Educational scoring may require trained raters applying a validated rubric.
For interpretive research tasks, there may be multiple defensible answers rather than one indisputable reference response.
Researchers should acknowledge such ambiguity and avoid treating one researcher's judgment as an infallible standard.
Where appropriate, multiple reviewers, explicit coding criteria, adjudication procedures, and documentation of disagreement can strengthen the reference process.
Evaluators should ideally be blinded to model version when that information could influence their judgments.
Which Performance Metrics Should Researchers Compare?
The metrics should reflect the system's intended use and the errors that matter.
| Research Task |
Potential Metrics |
Important Qualification |
| Binary classification |
Sensitivity, specificity, precision, recall, F1 score |
Class prevalence and error consequences affect interpretation |
| Multicategory classification |
Confusion matrix, macro-F1, category-specific recall |
Aggregate scores may conceal weak minority-category performance |
| Numerical prediction |
Mean absolute error, root mean squared error, calibration where relevant |
Different metrics emphasize different error properties |
| Ordinal scoring |
Weighted agreement, score differences, rubric-specific error analysis |
Ordinal categories should not automatically be treated as interval measurements |
| Information extraction |
Field-level precision, recall, exact-match accuracy, source fidelity |
Different extraction fields may have different error consequences |
| Generative responses |
Expert-rated correctness, completeness, unsupported-claim rates, task success |
Human evaluation criteria and rater reliability require attention |
One metric rarely captures every relevant performance characteristic.
For consequential research applications, subgroup performance and specific failure modes may be more informative than small changes in overall accuracy.
How Can Researchers Compare Two Classifiers Statistically?
For paired binary correctness outcomes, McNemar's test can assess whether the two systems differ in their marginal probability of producing a correct classification on the same cases.
The test focuses on discordant pairs: observations correctly classified by one model but incorrectly classified by the other.
It does not use the number of cases both models classify correctly or both classify incorrectly as direct evidence of a difference.
McNemar's test addresses a particular statistical question. It does not establish that a difference is practically important, nor does a nonsignificant result establish equivalence.
Researchers should also report the estimated difference in performance with an appropriate confidence interval and examine which errors changed.
For sensitivity or specificity comparisons, the relevant paired analysis should be conducted within the appropriate reference-standard subgroup.
How Should Researchers Evaluate Changes in Continuous or Ordinal Scores?
For continuous measurements, paired differences can reveal systematic offsets between versions. Agreement analysis may also be useful when the question is whether the systems can be used interchangeably.
Bland and Altman's work explains why correlation alone is insufficient for assessing agreement between measurement methods.
For ordinal rubric scores, researchers should consider the ordering and substantive meaning of categories. Weighted agreement measures or analyses of category transitions may be more appropriate than treating the scores as automatically continuous.
In both cases, comparison against an independent reference standard can help determine whether the updated model improved or worsened measurement quality.
How Do Researchers Distinguish Statistical Significance From Materiality?
Statistical significance concerns evidence against a specified null hypothesis under a particular statistical model.
Materiality concerns whether a difference is large enough to affect the research decision.
With a very large evaluation dataset, a small performance difference may be statistically detectable but practically unimportant. With a small dataset, a consequential difference may remain statistically uncertain.
Researchers should therefore establish a substantively justified threshold for meaningful change and interpret confidence intervals in relation to that threshold.
Where the objective is to demonstrate that performance remains sufficiently similar, formal equivalence or noninferiority approaches may be appropriate.
Lakens' work on equivalence testing explains why failing to reject a conventional difference test does not establish that two systems perform equivalently.
How Does Equivalence Testing Apply to AI Model Updates?
Equivalence testing evaluates whether the evidence supports ruling out differences larger than a predefined acceptable range.
For example, researchers might decide before testing that a change in a particular error rate exceeding two percentage points would be unacceptable for their intended application.
They would then select an appropriate equivalence procedure and sample size to determine whether the uncertainty around the estimated difference is sufficiently narrow.
The two-percentage-point threshold in this illustration is hypothetical. It is not a universal standard for AI evaluation.
The margin should be justified according to the task, risks, existing evidence, and intended interpretation.
For asymmetric consequences, noninferiority testing may be more appropriate when the central concern is preventing unacceptable deterioration rather than demonstrating similarity in both directions.
How Many Evaluation Cases Are Needed?
There is no universal sample size for comparing AI model versions.
Sample-size planning depends on the primary metric, expected performance, correlation between paired outcomes, acceptable difference, desired precision, and statistical power where hypothesis testing is used.
For rare but consequential errors, a large overall sample may still contain too few relevant cases to support precise conclusions.
Researchers should plan the evaluation around the decision it must support rather than choosing a convenient number of prompts.
When several observations come from the same participant, document, or institution, statistical dependence may also need to be accounted for.
Should Researchers Run Each Prompt More Than Once?
Repeated runs can be useful when the system produces variable outputs.
They allow researchers to estimate within-version variability and investigate whether differences between versions exceed ordinary variation.
However, repeated responses to the same input are not equivalent to independent observations from the target research population.
Analyses should account for the nesting of repeated outputs within cases rather than treating every generated response as an independent research case.
The number of repetitions should reflect the task, expected variability, and resources available.
What if the Earlier Model Version Is Unavailable?
If the original system cannot be rerun, researchers may use archived outputs, evaluation logs, and reference judgments to construct a retrospective comparison.
This approach can be informative when the original inputs and scoring procedures were preserved.
However, differences in evaluation timing, configuration, or sampling may limit causal attribution to the model update.
When the original outputs are incomplete or the evaluation conditions cannot be reconstructed, researchers should avoid claiming a precise estimate of version-related change.
They can still assess whether the updated system meets current performance requirements.
This distinction matters when researchers cannot lock an AI system to a specific version.
How Should Researchers Interpret a Mixed Pattern of Improvements and Deteriorations?
An update may improve some metrics while worsening others.
For example, a classifier might achieve higher overall accuracy but lower sensitivity for an uncommon category. Whether this represents an acceptable change depends on the intended research use.
Researchers should avoid reducing such findings to a single declaration that the newer model is better or worse.
Instead, report the relevant trade-offs and identify which performance characteristics matter most for the research decision.
This is particularly important when deciding whether outputs generated by different versions can be combined.
How Should Researchers Document a Model Update Evaluation?
A transparent report should identify the compared systems, evaluation dates, task definitions, datasets, reference standards, metrics, and statistical procedures.
It should also describe prompt configurations, repeated-run procedures where relevant, uncertainty estimates, subgroup analyses, and predefined criteria for material change.
Researchers should disclose any limitations in version identification or control over the surrounding system.
The National Institute of Standards and Technology's AI Risk Management Framework emphasizes measurement, validation, documentation, and monitoring of AI risks in their operational contexts.
Such documentation supports defensible decisions when a model changes during an ongoing study and helps other researchers interpret the evaluation after further updates occur.
