03 · What You Need to Know
Designing Fair and Interpretable Human-AI Performance Studies
What does fairness mean in a human-AI comparison?
In this context, fairness concerns whether the experimental conditions permit a meaningful interpretation of relative performance.
It does not necessarily mean that humans and AI must receive identical instructions, operate through identical interfaces, or use the same amount of time.
Instead, researchers should determine which differences are intrinsic to the systems being compared and which differences introduce avoidable confounding.
For example, an AI model may process information rapidly, while human researchers may rely on professional judgment, external references, and deliberation.
Whether these differences should be controlled or preserved depends on the research objective.
A study examining unaided factual recall differs from one examining the quality of complete research workflows.
What should researchers decide before conducting the comparison?
The evaluation should begin with an explicit statement of what is being compared.
| Comparison objective |
Appropriate focus |
Important design choice |
| Unaided capability |
Performance without external assistance |
Control available task information and tools |
| Realistic workflow performance |
Quality and efficiency under ordinary working conditions |
Allow justified workflow-specific resources |
| Human-AI collaboration |
Performance of people using AI assistance |
Compare with an appropriate human-only workflow |
| Professional competence |
Performance on tasks representative of professional practice |
Select relevant experts and authentic tasks |
| Time-constrained performance |
Quality achieved within defined deadlines |
Establish comparable and justified timing conditions |
These designs answer different questions. Researchers should not treat their findings as interchangeable.
How should human participants be selected?
The human comparison group should represent the population relevant to the intended claim.
If the study concerns professional research methodology, participants should possess qualifications and experience appropriate to that domain.
If the objective concerns educational support for novice researchers, undergraduate or postgraduate students may provide a relevant comparison.
Researchers should describe recruitment procedures, inclusion criteria, professional experience, and relevant training.
Convenience samples may be acceptable for exploratory studies, but their limitations should be acknowledged.
The selection of an appropriate human comparator is central to the validity of any relative performance claim.
Why is comparing AI with novices potentially misleading?
Comparing AI with novices is not inherently invalid. It becomes misleading when the findings are generalized to experienced professionals.
For example, a model outperforming undergraduate students on statistical terminology may demonstrate strong performance relative to introductory knowledge.
It does not establish that the model can provide reliable statistical consultation or replace experienced methodologists.
Researchers should distinguish novice, intermediate, and expert performance when those distinctions are relevant.
The choice between expert and novice comparators should follow the research question rather than the expected ranking.
Should humans and AI receive exactly the same questions?
Using common evaluation items can strengthen comparability because differences in item difficulty are reduced.
However, the way information is presented may require adaptation.
For example, an AI model may receive a research abstract through a structured text prompt, while a human researcher reads the same abstract in a document interface.
These presentations need not be identical in format, but they should communicate equivalent substantive information when the study intends to isolate performance on the same task.
Researchers should avoid providing additional methodological hints, examples, or contextual information to only one condition without justification.
Task equivalence should be established through the underlying requirements rather than superficial similarity of presentation.
Should human researchers be allowed to use external resources?
The answer depends on whether the evaluation concerns unaided performance or normal professional practice.
Researchers ordinarily use reference materials, statistical software, databases, and other tools.
Prohibiting these resources may be appropriate for a controlled memory or knowledge test but may misrepresent realistic professional performance.
Conversely, allowing AI access to retrieval systems while restricting humans to memory creates an information-access difference that must be considered.
Researchers should explain whether human participants may use their normal tools and why that arrangement supports the intended inference.
Should AI systems be allowed retrieval and other tools?
AI tool access should be defined according to the evaluation objective.
When assessing an underlying model's unaided capabilities, external retrieval or code execution may be restricted.
When evaluating a complete AI application, such tools may be integral to the system's intended operation.
Researchers should document enabled tools, external information sources, and relevant configurations.
They should not attribute improvements from retrieval or software execution exclusively to the language model's internal capabilities.
The methodological implications of AI tool access in human comparisons should be considered before testing.
Must AI and humans receive identical prompts?
Not necessarily. Human instructions and machine prompts may differ in presentation while conveying equivalent task requirements.
For example, human participants may receive a written instruction sheet, while an AI model receives the same classification criteria through a system or user message.
Researchers should preserve the substantive requirements and avoid introducing unequal guidance.
When AI prompts are optimized, the development procedure should be documented.
Providing the AI with extensive examples and iterative feedback while giving humans only brief instructions may create an advantage unrelated to the intended capability comparison.
Researchers should consider whether the preparation and task familiarization provided to each condition are appropriate and comparable.
How should time limits be established?
Time limits should correspond to the research question.
If the objective is to compare accuracy under a fixed deadline, a common time limit may be justified.
If the objective is to compare normal professional workflows, the study may instead allow each condition to complete the task and record elapsed time.
Researchers should avoid interpreting AI generation time as equivalent to the completion of a verified professional output.
Where human review is required, the combined workflow time should be included.
The decision concerning time limits in human-AI studies should be made before examining the results.
How can evaluator bias affect the comparison?
Evaluators may hold expectations about whether AI or human responses are more accurate, detailed, or reliable.
These expectations can influence subjective scoring, particularly for open-ended answers.
Researchers can reduce this risk by removing source-identifying information, randomizing response order, and applying explicit rubrics.
However, complete blinding may be difficult when responses contain recognizable stylistic features.
Evaluators should be instructed to judge substantive quality rather than infer the source of the response.
Where feasible, blinded evaluation procedures can strengthen the credibility of the comparison.
Should human and AI responses use the same scoring rubric?
Generally, yes, when the study compares performance on the same outcome.
A common rubric helps prevent one condition from being judged by more demanding criteria.
For example, both human and AI-generated research feedback might be assessed for factual correctness, methodological relevance, completeness, and consequential errors.
Researchers should avoid rewarding AI responses simply for being longer or more polished.
Likewise, concise human responses should not be penalized if they satisfy the substantive task requirements.
The rubric should be established independently of the observed performance differences and validated where appropriate.
How should the reference standard be established?
A reference standard should provide an independent basis for judging the evaluated outputs.
For factual questions, independently verified answers may be appropriate.
For professional judgments, a structured rubric or expert-adjudicated standard may be necessary.
The reference standard should not automatically privilege the preferences or output conventions of either condition.
For example, if experts establish the reference answers, their role as evaluators should be distinguished from any role as human comparison participants.
Researchers should document how disagreements and ambiguous cases were handled.
Should human participants complete the same number of tasks as AI?
Comparable task exposure is generally desirable, but the appropriate design depends on the unit of analysis.
An AI model may process hundreds of items rapidly, while assigning the same workload to each human participant could introduce fatigue.
Researchers may distribute items across participants using a balanced design while ensuring adequate coverage of the evaluation dataset.
Such designs require statistical analysis that accounts for which participants completed which items.
Researchers should not treat every response as independent when multiple observations come from the same participant or evaluation item.
Task assignment should also avoid systematically giving one condition easier questions.
How should learning, fatigue, and order effects be controlled?
Human participants may improve through practice or become fatigued during prolonged testing.
AI responses may also depend on conversation history or prior context when tasks are administered sequentially.
Researchers should consider randomizing item order, balancing task sequences, and using fresh conversation contexts where appropriate.
Practice effects may be addressed through familiarization tasks or other justified procedures.
These controls should reflect the intended application rather than automatically eliminating every difference between human and AI operation.
Order effects should be documented when they could influence the interpretation of comparative performance.
How should researchers analyze human-AI performance statistically?
The statistical method should reflect the study design.
When AI and humans answer the same items, their outcomes are paired at the item level.
When multiple humans contribute responses, observations may also be clustered within participants.
For binary outcomes, paired tests or hierarchical logistic models may be appropriate depending on the design.
For rubric-based scores, researchers may use suitable mixed-effects models or other procedures accounting for repeated observations.
Confidence intervals and effect sizes should accompany inferential comparisons where appropriate.
Researchers should distinguish a statistically supported advantage from a practically meaningful improvement.
Why should errors be examined separately?
Two conditions can achieve similar average accuracy while making different kinds of mistakes.
For example, AI may produce unsupported references, while humans may omit less familiar methodological details.
These errors may differ in severity and detectability.
Researchers should classify relevant error types and examine whether the distribution differs between conditions.
Such analysis may reveal that one condition is preferable for particular tasks despite similar overall performance.
For consequential applications, separate analysis of AI and human errors can provide more useful evidence than average accuracy alone.
Should human-AI collaboration be included?
When AI is intended to assist rather than replace researchers, a collaboration condition may be essential.
For example, a study could compare researchers working independently with researchers using AI-generated methodological suggestions.
The evaluation might measure final correctness, completion time, error detection, and the amount of correction required.
However, human-AI collaboration should not automatically be assumed to outperform either condition independently.
AI assistance may improve some tasks while introducing automation bias or additional review burdens in others.
The collaborative workflow should be evaluated empirically using comparable final-output standards.
What ethical issues arise in human-AI comparison studies?
Studies involving human participants may require ethics review, informed consent, and appropriate protection of participant information, depending on institutional requirements and the research design.
Researchers should consider whether participants understand how their performance will be compared and reported.
Professional reputation or employment consequences may arise if individual results are identifiable.
Where appropriate, findings should be reported in aggregate form and avoid unnecessary identification of participants.
Researchers should also consider workload fairness, reasonable compensation, and whether the testing conditions impose avoidable burdens.
Ethical research design should not be sacrificed merely to create a dramatic human-versus-AI comparison.
What conclusions can a fair comparison support?
A well-designed comparison can establish relative performance under the tested conditions.
For example, researchers may conclude that an AI model achieved higher observed classification accuracy than a specified group of human participants on a particular dataset.
If the statistical analysis supports it, the study may establish superiority on that outcome.
However, the findings do not automatically establish broader professional competence, causal explanations for performance differences, or interchangeability.
Researchers should distinguish the comparative result from the wider question of what better-than-human AI performance means.
