03 · What You Need to Know
Establishing a Credible Human Performance Baseline
What is a human comparator?
A human comparator is an individual or group whose performance is measured as a reference for evaluating an AI system on a defined task.
For example, researchers may compare an AI model's classification of research abstracts with classifications produced by postgraduate students, experienced researchers, or methodology specialists.
The human comparator provides an empirical performance baseline. It should not be confused with the reference standard used to determine whether a response is correct.
Human comparator
The person or population performing the task against which AI performance is compared.
Reference standard
The independently established evidence or criteria used to evaluate the correctness or quality of outputs from both conditions.
For example, participating researchers and an AI model might independently classify the same abstracts, while a separately adjudicated set of methodological labels provides the reference standard.
Why does comparator selection matter?
Different human populations possess different levels and types of task-relevant expertise.
An experienced researcher may recognize methodological ambiguity that a novice overlooks. A specialist in qualitative inquiry may perform well on interpretive tasks but less confidently on advanced statistical questions outside their area of expertise.
Consequently, human performance should not be treated as one fixed quantity.
The literature on human-AI evaluation has emphasized the need to document human baselines rigorously rather than assuming that a convenient participant group represents general human ability.
Researchers should therefore select comparators according to the intended inference and explain why the chosen population is relevant.
Which human comparator groups might be appropriate?
| Comparator group |
Appropriate research objective |
Important limitation |
| Undergraduate students |
Evaluating AI relative to introductory knowledge or learner performance |
Does not represent professional expertise |
| Postgraduate students |
Evaluating AI relative to developing research competence |
Training and experience may vary considerably |
| Early-career researchers |
Evaluating performance on tasks typical of early research practice |
Career stage does not guarantee consistent expertise |
| Experienced researchers |
Comparing AI with established professional research practice |
Expertise may be concentrated in particular disciplines |
| Methodological specialists |
Testing performance on specialized methodological decisions |
May not represent ordinary research workflows |
| Professional teams |
Comparing AI with collaborative institutional practice |
Team outcomes depend on coordination and available resources |
The most suitable comparator is not necessarily the most highly qualified group. It is the group that provides the most meaningful baseline for the question being investigated.
Should the human comparator always be an expert?
No. Expert comparison is necessary for some claims but may be unnecessary or inappropriate for others.
If the study examines whether AI can support introductory research education, comparison with learners may be directly relevant.
If the study claims that AI can perform at the level of professional researchers, a relevant professional comparator is needed.
Researchers should distinguish claims about novice-level performance from claims about specialized competence.
The choice between experts and novices in AI performance studies should therefore be determined by the research objective.
How should researchers define expertise?
Expertise should be operationalized through criteria relevant to the evaluated task.
Academic rank, degree attainment, publication count, and years of experience may provide contextual information, but none independently establishes competence in every research activity.
For example, a senior researcher specializing in qualitative inquiry may not be an appropriate expert comparator for advanced Bayesian modeling.
Researchers may consider task-specific training, demonstrated competence, professional responsibilities, relevant publications, or independently assessed proficiency.
Where feasible, a preliminary competency assessment can provide additional evidence about the actual capabilities of participating humans.
Expertise categories should be defined before examining performance outcomes to avoid post hoc classification that favors a preferred result.
Can professional titles be used as evidence of expertise?
Professional titles can support participant description, but they should not be treated as definitive evidence of task competence.
A professor may possess extensive teaching experience but limited experience with a specialized analytical method.
Similarly, a postgraduate student may have substantial technical expertise in a narrow domain.
Researchers should distinguish general professional status from task-specific proficiency.
When titles are used as eligibility criteria, the study should explain why they reasonably represent the capability being evaluated.
Should researchers use one human comparator or several?
One comparator population may be sufficient for a narrowly defined research question.
However, including several experience levels can help establish how AI performance relates to a broader distribution of human competence.
For example, a study might compare AI with postgraduate students and experienced methodology specialists.
This design can reveal whether AI exceeds developing competence while remaining below specialist performance.
Including additional groups also increases recruitment requirements, statistical complexity, and the need for adequate subgroup sample sizes.
Researchers should therefore include multiple groups when they contribute substantively to the intended inference, not merely to make the study appear comprehensive.
How many human participants are needed?
There is no universal minimum number of human comparators.
Sample-size requirements depend on the expected performance difference, variability among participants, number of evaluation items, study design, and desired statistical precision.
A study involving five experts who each evaluate hundreds of items does not necessarily provide the same information as a study involving hundreds of participants who each evaluate only a few items.
Both the number of participants and the distribution of tasks across them matter.
Researchers should conduct appropriate sample-size or precision planning when making inferential comparisons.
Repeated responses from the same participant should not be treated as though they came from independent individuals.
Should researchers compare AI with average human performance?
Average human performance can provide a useful baseline, but its meaning depends on how the average is calculated.
A mean across all human responses may differ from the mean of individual participant accuracy scores when participants complete unequal numbers of items.
Researchers should specify whether they report response-weighted accuracy, participant-level mean accuracy, median performance, or another summary.
The distribution of human performance may also be informative.
A model may exceed the group average while performing below the strongest specialists.
Researchers should avoid interpreting a higher score than the human mean as evidence that AI outperforms every human participant.
How should human performance variability be reported?
Human participants may differ substantially in task performance.
Researchers can report relevant descriptive statistics, including participant-level means, medians, standard deviations, and ranges where appropriate.
However, these measures should be interpreted in relation to the study design and number of tasks completed by each participant.
When participants receive different items, observed score variation may reflect differences in task difficulty as well as individual competence.
Statistical models can account for participant and item variation when the data structure supports such analysis.
Reporting variability helps prevent one aggregate human score from being mistaken for a universal benchmark.
Should the same human participants serve as reference evaluators?
Using comparison participants to establish the reference standard can introduce circularity or conflicts of interest.
For example, if participating researchers collectively determine the correct classifications, their own responses may receive an advantage over AI outputs.
Where feasible, the reference standard should be established independently of the evaluated comparison conditions.
If expert judgment is necessary, a separate adjudication group may be appropriate.
Researchers should document any overlap between comparison participants and reference evaluators and explain its implications.
The principles of reference standard development remain relevant.
Should humans and AI complete the same evaluation items?
Using common items can improve comparability by reducing differences in task difficulty.
However, assigning every item to every human participant may create excessive workload or fatigue.
Researchers can use balanced incomplete designs in which participants complete subsets of items while maintaining sufficient overlap for comparison.
Such designs require careful task allocation and statistical analysis.
Researchers should avoid systematically assigning easier items to one condition.
When human and AI responses are evaluated on different datasets, conclusions about relative performance require additional justification.
How should familiarity and preparation be handled?
Human participants may require instructions, examples, or practice trials to understand the evaluation procedure.
AI systems may similarly receive task definitions, examples, or prompt optimization.
Researchers should document the preparation provided to both conditions.
Extensive prompt engineering for AI combined with minimal human task familiarization may introduce an avoidable imbalance.
Conversely, giving human participants extensive coaching while evaluating AI through an untested prompt may also distort the comparison.
Preparation should be justified by the study's objective rather than designed to favor one condition.
Should human comparators use their normal professional tools?
For realistic workflow comparisons, access to ordinary professional resources may be appropriate.
Researchers commonly use databases, statistical software, methodological references, and collaborative consultation.
Restricting these resources can create an artificial human baseline.
However, unaided human performance remains a legitimate research condition when the objective specifically concerns unaided capability.
The study should distinguish the two designs.
The implications of human tool access should be considered alongside the tools available to the AI system.
How should statistical comparisons account for multiple humans?
When multiple participants evaluate multiple items, observations may be correlated within participants and within evaluation items.
For binary correctness outcomes, a generalized linear mixed model with appropriate participant and item effects may be useful.
Other approaches, including cluster-aware resampling, may be appropriate depending on the design.
The analysis should reflect whether the same items were evaluated by humans and AI, how tasks were allocated, and whether the AI generated repeated responses.
Researchers should report effect sizes and uncertainty rather than relying solely on whether one average is numerically larger.
What ethical considerations apply to human comparators?
Human performance studies may involve identifiable information about professional competence.
Researchers should consider appropriate ethics review, informed consent, confidentiality, and participant welfare.
Participants should understand the study procedures and how their performance data may be reported.
Individual scores should not be disclosed unnecessarily, particularly when results could affect professional reputation or employment.
Recruitment and compensation should also reflect reasonable participant burden.
Researchers should avoid treating human participants merely as convenient benchmark data.
What should be reported about the comparator population?
A transparent report should identify recruitment procedures, eligibility criteria, participant numbers, expertise definitions, task assignments, permitted resources, and relevant demographic or professional characteristics where justified.
Researchers should also report how many items each participant completed, how performance was aggregated, and how missing or invalid responses were handled.
The statistical analysis should account for the design's dependence structure.
These details are essential when interpreting claims that AI performs better than humans.
Without them, readers may be unable to determine what the reported human baseline actually represents.
