03 · What You Need to Know
Choosing Appropriate Human and Software Comparators in AI Research
What is a comparator in an AI performance study?
A comparator is the person, system, procedure, or condition against which the performance of an AI system is assessed.
Comparators provide context for interpreting performance. An AI model achieving 85% accuracy may appear impressive in isolation, but the significance changes if an existing software tool achieves 95% accuracy on the same task.
Similarly, an AI system may perform better than novice researchers but less accurately than experienced specialists.
The comparator therefore helps establish what the AI system contributes relative to available alternatives.
Researchers should distinguish the comparator from the reference standard used to determine correctness. A comparator is another performer or workflow. A reference standard provides the basis for judging performance.
Comparator
The alternative person, technology, or procedure whose performance is evaluated alongside the AI system.
Reference standard
The independently defensible criteria or evidence used to judge whether the resulting outputs satisfy the task requirements.
For example, human researchers and an AI model may both classify the same abstracts, while independently adjudicated methodological labels serve as the reference standard.
When should AI be compared with humans?
Human comparison is appropriate when the research question concerns tasks normally performed by people.
Examples include interpreting research findings, screening literature, providing educational feedback, assessing qualitative responses, and identifying methodological weaknesses.
However, the relevant human comparator depends on the intended application.
If AI is proposed to assist experienced researchers, comparing it exclusively with students who have limited methodological training may provide little evidence about its suitability for professional research workflows.
Conversely, if the intended application is introductory research education, student performance may be directly relevant.
The study should explain why the selected human group represents the appropriate comparison population.
The broader question of who should count as the human comparator is particularly important when researchers make claims about expert-level AI performance.
Should researchers compare AI with experts or novices?
Neither group is automatically the correct comparator.
Expert comparisons are appropriate when the study examines whether AI can perform tasks requiring specialized professional competence.
Novice comparisons may be appropriate when the research concerns educational support, entry-level performance, or the potential effects of AI on learners.
In some studies, including both groups may help establish a more informative performance context.
| Human comparator |
Appropriate research question |
Important limitation |
| Novices or students |
Can AI perform tasks at or above an introductory level? |
Results do not establish equivalence with experienced professionals |
| Experienced practitioners |
How does AI compare with routine professional performance? |
Experience and specialization may vary considerably |
| Subject-matter experts |
Can AI approach performance on specialized tasks? |
Expert performance is not necessarily uniform or error-free |
| Mixed-experience groups |
How does comparative performance vary by expertise? |
Subgroup sample sizes and task relevance require attention |
Researchers should avoid using labels such as expert or novice without explaining the qualifications or criteria underlying those categories.
The distinction between expert and novice human comparisons should follow the study's intended inference rather than whichever group makes the AI system appear more impressive.
When should AI be compared with existing software?
Software comparison is appropriate when the AI system is proposed as an alternative to an established computational approach.
For example, an AI-based research classification system might be compared with conventional machine-learning classifiers, rule-based software, or other tools designed to perform the same task.
A generative AI system used for information extraction might be compared with existing extraction pipelines rather than only with manual extraction.
Such comparisons help determine whether the AI approach provides an improvement over available technological alternatives.
However, the software comparator should be relevant and competently implemented.
Comparing a sophisticated AI system with an intentionally weak or poorly configured baseline may produce a technically correct result that is nevertheless misleading about practical value.
What makes a software baseline credible?
A credible software baseline should represent a meaningful alternative for the intended task.
Researchers should consider whether the software is established in the relevant field, whether it supports the task, and whether its configuration reflects reasonable practice.
Where appropriate, the comparison may include a simple baseline and a stronger established method.
A simple baseline can reveal whether the AI system performs better than a minimal approach. A stronger baseline helps establish whether it improves upon methods that researchers might realistically adopt.
The baseline should not be selected solely because it is easy for the AI system to outperform.
Researchers should document software versions, relevant parameters, training procedures, and any tuning performed.
Why might comparing AI with both humans and software be useful?
Human and software comparisons answer different questions.
A human comparator establishes how AI performance relates to the abilities or workflows of people performing the task.
A software comparator establishes how AI performs relative to computational alternatives.
Including both can reveal whether an apparent advantage is genuinely distinctive.
For example, an AI system might outperform novice researchers but perform similarly to an established classification algorithm. That result would support a different interpretation from one in which AI substantially outperforms both comparators.
However, adding comparators increases study complexity, data requirements, and analytical decisions. Researchers should include them because they are relevant to the research question, not merely to make the evaluation appear more comprehensive.
Should the study include a human-AI collaboration condition?
When AI is intended to assist rather than replace human work, a human-AI collaboration condition may be more relevant than comparing AI alone with humans alone.
For example, an educational institution considering AI-assisted assessment may need to know whether instructors using AI produce better feedback than instructors working without it.
In such a study, possible conditions include human-only work, AI-only performance, and human-AI collaboration.
The collaboration condition can reveal whether AI adds value to the existing workflow.
It can also reveal additional burdens, such as the time required to verify AI-generated claims or correct inappropriate suggestions.
Researchers should define how humans are permitted to interact with AI and whether the workflow resembles the intended real-world use.
Should humans and AI receive identical information?
Information access can strongly influence comparative performance.
If the AI system receives detailed reference documents while human participants must answer from memory, the resulting difference may reflect unequal information access.
Conversely, preventing professionals from using tools they ordinarily rely on may create an unrealistic human comparison.
The appropriate arrangement depends on whether the study seeks to isolate unaided capability or evaluate realistic workflows.
For controlled capability comparisons, equivalent task information is generally desirable.
For deployment-oriented comparisons, allowing each condition to use its normal resources may be more appropriate, provided the study clearly identifies those differences.
The methodological question of whether humans and AI should receive the same information should be resolved before testing.
Should human participants be allowed to use their normal tools?
Often, yes, when the study evaluates realistic professional performance.
Researchers, clinicians, programmers, and other professionals frequently use reference materials, software, and established procedures as part of competent practice.
Comparing an AI system with tool-assisted capabilities against humans who are prohibited from using ordinary professional resources may exaggerate the AI's practical advantage.
However, unaided human performance can still be a legitimate research condition when the objective is explicitly to compare unaided capabilities.
Researchers should distinguish these designs and avoid interpreting one as evidence for the other.
Whether human participants should be allowed their normal tools depends on the study's target use case.
Should AI systems be allowed external tools?
AI models may use retrieval systems, code execution, databases, or other external resources.
These capabilities can materially affect performance.
For a model-level comparison, researchers may need to control tool access to isolate the behavior of the underlying model.
For a complete-system comparison, tool use may be part of the system being evaluated.
The distinction should be explicit.
A retrieval-augmented AI system outperforming an unaided human does not necessarily establish that the language model itself possesses superior internal knowledge.
The result may instead demonstrate the effectiveness of the complete retrieval-supported workflow.
Should humans and AI have the same time limits?
Equal time limits can support some controlled comparisons, but they are not always the most meaningful arrangement.
An AI system may generate an answer in seconds, while a human expert may require several minutes to examine the evidence carefully.
If the objective is to compare performance under a fixed deadline, equal time limits may be justified.
If the objective is to evaluate accuracy under normal working conditions, forcing identical time limits may distort the comparison.
Researchers should distinguish speed from correctness and consider reporting both.
The question of time limits in human-AI performance studies should be addressed according to the intended practical application.
How should performance be measured across comparators?
The same relevant outcomes should generally be assessed across comparison conditions.
For a classification task, this might include accuracy, precision, recall, and the distribution of errors.
For open-ended feedback, a common rubric might assess factual accuracy, relevance, completeness, and the severity of consequential mistakes.
Researchers should avoid scoring AI outputs using one standard and human outputs using another unless the difference is explicitly justified by the research objective.
Human evaluators may also be influenced by knowing whether an answer came from AI or a person. Blinded evaluation can reduce this risk when feasible.
When the study uses an automated evaluator, the scoring procedure should be validated to ensure that it does not systematically favor particular response styles.
How should researchers interpret differences in accuracy?
Observed accuracy differences should be considered alongside statistical uncertainty and the consequences of errors.
Suppose AI achieves 88% accuracy, experienced researchers achieve 90%, and established software achieves 86%.
These percentages alone do not establish that the human group is definitively superior or that AI provides a meaningful improvement over software.
The analysis should consider sample size, pairing by evaluation item, repeated observations, and relevant uncertainty intervals.
Researchers should also examine error types. Two systems with similar average accuracy may differ substantially in how often they make consequential mistakes.
For deployment decisions, speed, cost, and the amount of human oversight required may also influence the preferred option.
Does similar performance mean AI and humans are interchangeable?
No.
Similar average scores do not establish that two performers can safely or effectively substitute for one another.
AI and human experts may make different errors, respond differently to unfamiliar cases, and require different forms of oversight.
Furthermore, a failure to detect a statistically significant difference does not establish equivalence.
Equivalence or noninferiority claims require appropriate margins, study designs, and statistical analyses.
Researchers should therefore avoid concluding that AI can replace professionals merely because its average score is close to theirs.
The distinction between similar accuracy and genuine interchangeability is particularly important for consequential applications.
When is a human or software comparison unnecessary?
Not every AI evaluation requires an external comparator.
A descriptive study may legitimately investigate the error distribution, response stability, or prompt sensitivity of one AI system without comparing it with humans or existing software.
Likewise, a methodological study may focus on how changes in prompt formulation influence performance.
However, claims that AI is better, faster, more accurate, or more efficient than existing alternatives require evidence from an appropriate comparison.
The comparator should follow from the claim, not the other way around.
