How Do Researchers Compare AI Models Fairly?

How Can You Tell Whether One AI Model Really Performs Better Than Another?

You want to compare two generative AI models for a research task. Both receive the same set of questions. One produces more correct answers, so you conclude that it is the better model.

But what if one model had access to external sources while the other relied entirely on its internal knowledge? What if their response settings differed, or one model received instructions better suited to its capabilities?

Even when researchers use identical questions, the comparison may not be methodologically fair.

A scientifically defensible AI comparison requires researchers to establish what is being compared, which conditions should be controlled, how performance will be measured, and which conclusions the evidence can support.

Fair Comparisons Require Equivalent Opportunities, Not Necessarily Identical Configurations

Designing a Scientifically Defensible AI Model Comparison

First, decide what you are actually comparing

The phrase "AI model comparison" can refer to several different research objectives.

A researcher might want to compare the intrinsic capabilities of two language models, the performance of two commercial chat applications, or the effectiveness of two AI-assisted workflows.

These are not interchangeable.

For example, comparing two commercial chat interfaces with different retrieval capabilities may be appropriate when the research question concerns which complete service better supports researchers. It would be less appropriate to interpret the results as an isolated comparison of the underlying language models.

The evaluation target, sometimes called the estimand in statistical research, should be specified before data collection.

Should AI models receive the same test questions?

Generally, yes. When the objective is to compare performance on a shared task domain, the models should ordinarily be evaluated using the same underlying test items.

Using different questions for each model can introduce differences in difficulty, content, and domain coverage that complicate interpretation.

However, identical test items do not necessarily require identical prompt formatting.

Different systems may use different message structures, supported modalities, or instruction conventions. A carefully controlled adaptation may be necessary to communicate the same task effectively.

The methodological question is whether the adaptation preserves the intended task and information content without introducing an unjustified advantage.

Researchers should distinguish between maintaining the same evaluation questions and requiring different AI models to receive identical prompts.

When is identical prompting fair, and when might it be misleading?

Identical prompting is particularly useful when researchers want to examine how different models respond to exactly the same textual input.

It reduces one source of experimental variation and makes the procedure relatively straightforward to document.

Nevertheless, models may respond differently to the same prompt because of their training, instruction-following behavior, context handling, and interface conventions.

A prompt that is effective for one model may be poorly suited to another.

Model-specific prompt optimization can therefore be legitimate when the study evaluates achievable performance under a defined optimization procedure. However, allowing extensive optimization for one model while providing little or none for another would introduce an avoidable asymmetry.

Neither approach is universally superior. The appropriate design follows from the question the researcher wants to answer.

Researchers should also examine whether prompt wording may influence the study's conclusions, particularly when differences between models are small.

Should models have access to the same tools and information?

For comparisons intended to isolate model capabilities, unequal access to external information can create substantial confounding.

Consider a study evaluating answers to recent scientific questions. If one model can search the internet while another cannot, superior performance may reflect access to current sources rather than differences in internal knowledge.

Similarly, retrieval-augmented generation, code execution, file analysis, and other external tools can influence performance.

Researchers should decide whether tools are part of the system being evaluated or factors that should be controlled.

For a model-only comparison, comparable information access is generally desirable. For an application-level comparison, retaining the native capabilities of each service may be appropriate, provided the resulting conclusions concern the complete systems.

The same principle applies to context windows, uploaded documents, and any additional information supplied to a model.

How should researchers handle different model settings?

Generation settings may influence output quality, consistency, speed, and cost.

Relevant settings can include temperature, top-p sampling, maximum output length, reasoning effort, random seeds, and other provider-specific controls.

However, parameters with similar names do not necessarily have identical implementations across model families.

Setting temperature to the same numerical value for two models does not establish that their output distributions are equivalent.

Some reasoning-oriented systems also expose controls that are unavailable in other models.

Researchers should document available settings and justify how they were selected. Where direct equivalence is impossible, the limitation should be acknowledged rather than concealed behind superficially identical parameter values.

How should the evaluation dataset be selected?

A fair comparison requires test data appropriate to the intended application.

If the research concerns AI support for educational research, a benchmark consisting entirely of general trivia questions would provide limited evidence about the systems' ability to classify study designs, interpret methodological descriptions, or evaluate research arguments.

Dataset selection should reflect the task domain, difficulty distribution, language requirements, and relevant user population.

Researchers should consider whether the test items are representative of the situations to which they intend to generalize.

Benchmark contamination is another concern. Models may have encountered publicly available benchmark items or closely related material during training or development. High performance on familiar questions may not reflect the same capability as performance on genuinely unseen tasks.

Where feasible, researchers can use independently developed or carefully held-out evaluation items. However, newly written questions are not automatically free from conceptual overlap with training material.

Why should researchers establish a reference standard?

Comparative performance claims require a defensible basis for determining whether outputs are correct or otherwise satisfactory.

For factual questions, the reference standard might be a verified answer supported by authoritative sources. For classification tasks, it may involve expert-developed labels and an adjudication procedure.

For open-ended responses, researchers may need a structured rubric covering factual accuracy, completeness, relevance, and other task-specific dimensions.

The reference standard should be established independently of the models' preferred answers whenever feasible.

Researchers should avoid treating agreement with one AI model as proof that another model is correct.

The selection and validation of an appropriate AI evaluation reference standard deserve particular attention when the task involves expert judgment or multiple defensible answers.

Which performance metrics should be compared?

Accuracy may be central to some studies, but it is not sufficient for every application.

The Stanford Center for Research on Foundation Models' Holistic Evaluation of Language Models (HELM) work illustrates the value of evaluating systems across multiple scenarios and dimensions rather than relying on a single aggregate score.

The relevant measures should follow from the research objective.

These measures should not be combined into an arbitrary overall score without a justified weighting scheme.

A model may be more accurate but slower, or faster but less reliable on difficult tasks. Such trade-offs may be more informative than a declaration that one system is universally superior.

How should repeated responses be handled?

Generative AI systems may produce different answers to the same prompt across repeated trials.

If one model is tested once per question while another receives several attempts and is scored using its best answer, the comparison is not balanced.

Researchers should specify whether the evaluation concerns first-response performance, average performance across repeated generations, or a defined multi-generation procedure.

Where repeated testing is appropriate, comparable sampling procedures should be applied to the systems under investigation.

The analysis should account for responses clustered within test items and distinguish variation between questions from variation between generations.

Further methodological decisions about nondeterministic AI responses should be made before examining which model appears to perform better.

How should researchers compare model performance statistically?

When models answer the same evaluation questions, their results are paired by test item. This pairing should generally be preserved in the analysis.

For example, if two models produce one binary correct-or-incorrect outcome for each shared question, McNemar's test may be appropriate for evaluating differences in paired success proportions.

For repeated generations or more complex outcomes, hierarchical models, paired bootstrap procedures, or other methods may be more suitable.

Statistical significance alone should not determine which model is preferable. Researchers should also report the magnitude of observed differences, uncertainty intervals, and practical consequences.

Small differences may be inconclusive, especially when the dataset is limited or evaluation scores depend on subjective judgments.

Should researchers blind evaluators to model identity?

When human evaluators judge response quality, knowledge of which model produced an answer may influence their assessments.

Researchers can reduce this risk by removing model identifiers, randomizing presentation order, and applying the same evaluation rubric to all outputs.

However, blinding may be imperfect if models have recognizable formatting or writing patterns.

Evaluators should also receive clear guidance on scoring criteria, and agreement between evaluators should be assessed when relevant.

For automated evaluation, researchers should examine whether the grading procedure favors particular response styles or model families.

How should model updates and testing dates be handled?

Commercial AI models and their surrounding applications can change over time.

Comparing one model in January with another in June may introduce temporal differences that complicate interpretation.

Where practical, researchers should test models during comparable periods and document exact available identifiers, dates, settings, and service conditions.

If one model changes during the evaluation, the affected observations may need to be separated or analyzed with appropriate qualifications.

Researchers should not assume that a commercial product name uniquely identifies an unchanged technical system.

Can a fair comparison still favor one model?

Yes. Fairness does not require equal performance.

A well-designed study may legitimately find that one model performs substantially better than another on a particular task.

The important requirement is that the difference can reasonably be attributed to the evaluated systems under the specified conditions rather than to an avoidable imbalance in the research design.

Even then, the conclusion should remain appropriately bounded.

Evidence that one model performs better on educational research classification does not establish that it is better at programming, medical reasoning, or every other task.

Comparing Two AI Models for Research Methodology Classification

The example illustrates why a shared dataset and paired analysis matter. Although Model A has a higher observed accuracy, the magnitude and uncertainty of the difference must be considered before making a strong comparative claim.

If the researcher later allows both models to generate multiple responses and selects a majority classification, that would constitute a different evaluation procedure requiring separate reporting.

Common Mistakes That Make AI Comparisons Misleading

Choosing a Fair AI Comparison Design for Your Research

Before selecting models or constructing prompts, identify the decision the comparison is intended to support.

A study intended to identify the most accurate model for a controlled classification task may require a different design from one intended to determine which AI service offers the most useful assistance to working researchers.

Researchers should also prespecify the main evaluation outcomes where feasible. Choosing the metric that most favors a particular model after examining the results can introduce selective reporting.

For broader studies, evaluating AI performance across relevant dimensions may provide a more informative assessment than relying on a single accuracy score.

Before Comparing AI Models in a Research Study

Frequently Asked Questions About Fair AI Model Comparisons

Fair AI Comparisons Depend on the Research Question and Experimental Design

How to Cite This Guide

This guide is intended to be read, shared, and used in research, teaching, and academic work. If you draw on its ideas, explanations, or other content, please acknowledge the source by citing the guide. Doing so gives appropriate credit and helps your readers locate the original resource.