03 · What You Need to Know
Designing a Valid and Interpretable AI Performance Evaluation
Begin with the task, not the metric
The first step is to identify what the AI system is expected to accomplish.
An AI model used to classify research abstracts performs a different task from one used to summarize journal articles, generate programming code, or provide feedback on student writing.
Each application requires different evidence of successful performance.
Researchers should specify the intended users, input characteristics, expected outputs, operating conditions, and consequences of errors.
For example, an AI system intended to support preliminary literature screening may be evaluated for its ability to identify potentially relevant records. A system intended to make final eligibility decisions without human review would require a more demanding assessment of missed relevant studies and other consequential errors.
The evaluation should reflect the actual claim being investigated rather than relying on whichever metric is easiest to calculate.
What exactly is being evaluated?
Researchers should distinguish the underlying model from the application or workflow through which it is used.
Model-level evaluation
Assesses model behavior under specified input, generation, and information-access conditions.
System-level evaluation
Assesses a complete application or workflow, potentially including retrieval, external tools, human review, and other components.
A system that combines a language model with document retrieval may outperform the same model operating without external information.
That improvement may reflect the retrieval process, the model's use of retrieved evidence, or their interaction.
Researchers should therefore describe the actual evaluated system and avoid attributing every observed result exclusively to the underlying model.
How should the evaluation dataset be selected?
The evaluation dataset should represent the tasks and conditions to which the findings are intended to apply.
For example, a study evaluating AI support for educational research methodology should include appropriate examples of the research designs, terminology, and methodological ambiguities encountered in that domain.
A dataset containing only straightforward textbook questions may overestimate performance on authentic research tasks.
Researchers should consider the distribution of task difficulty, disciplinary coverage, language, document length, and relevant subgroups.
Where possible, the dataset should be independent of prompt development and model selection.
Public benchmarks may be useful, but researchers should consider possible training-data contamination and whether the benchmark remains appropriate for the intended application.
What should count as the reference standard?
A reference standard provides the basis for judging whether an AI output is correct or otherwise satisfactory.
For factual tasks, it may consist of verified answers supported by authoritative evidence. For classification tasks, it may involve expert-assigned labels established through a documented procedure.
For open-ended tasks, the standard may be a structured evaluation rubric rather than one uniquely correct answer.
Reference standards are not automatically infallible. Expert judgments may differ, and some tasks admit multiple defensible interpretations.
Researchers should explain how the standard was developed, whether disagreements were resolved, and what limitations remain.
The choice of an appropriate reference standard for AI evaluation is especially important when model outputs involve complex reasoning or subjective assessment.
Which metrics should researchers use?
Performance measures should correspond to the outcome and the consequences of errors.
| Task |
Possible metrics |
Important consideration |
| Classification |
Accuracy, precision, recall, F1 score, confusion matrix |
Class imbalance and unequal error consequences |
| Numerical prediction |
Mean absolute error, root mean squared error, calibration measures where applicable |
Magnitude and distribution of prediction errors |
| Information retrieval |
Precision, recall, ranking metrics |
Consequences of missed or irrelevant records |
| Text summarization |
Factual consistency, coverage, relevance, human-rated quality |
Unsupported claims and omitted essential information |
| Code generation |
Functional test success, correctness, security-related assessments |
Whether code performs the intended task safely |
| Open-ended advice |
Rubric-based correctness, completeness, appropriateness, and error severity |
Multiple acceptable responses and evaluator judgment |
Not every metric should be applied to every task.
For example, classification accuracy can be misleading when one category dominates the dataset. A model that predicts the majority category for every item may achieve apparently high accuracy while performing poorly on less frequent categories.
Researchers should choose metrics that reveal the performance characteristics relevant to the intended use.
How should accuracy be calculated and interpreted?
For a classification task with one correct reference label per item, accuracy is the proportion of evaluated predictions that match the reference labels.
The result indicates that 84% of the evaluated predictions matched the reference labels under the tested conditions.
It does not establish that the model will achieve 84% accuracy on every future dataset or that the remaining errors are equally consequential.
Researchers should report uncertainty estimates where inferential claims are made and examine the distribution of errors across relevant task categories.
The meaning of accuracy in generative AI research may be more complicated when outputs are open-ended or partially correct.
How should partially correct AI answers be evaluated?
Many generative AI tasks do not produce responses that are simply correct or incorrect.
An AI-generated explanation may identify the correct statistical method but misunderstand one of its assumptions. A literature summary may accurately describe several findings while including an unsupported claim.
Binary scoring can conceal these distinctions.
Researchers may use a prespecified rubric that assigns scores to relevant dimensions or distinguishes fully correct, partially correct, incorrect, and nonresponsive outputs.
However, partial-credit rules should be justified and consistently applied.
Researchers should avoid creating scoring categories after examining which model benefits most from a particular interpretation.
The evaluation of partially correct AI answers requires attention to the consequences of specific errors, not merely the proportion of accurate statements.
Why should error types be analyzed separately?
Two models can achieve the same average accuracy while making substantially different kinds of errors.
For example, one model may frequently omit relevant information, while another produces detailed responses containing fabricated references.
Depending on the application, these errors may have different consequences.
Researchers should consider classifying errors according to task-relevant categories such as factual inaccuracy, unsupported claims, omission, incorrect classification, or failure to follow instructions.
The categories should be clearly defined, and the scoring procedure should be sufficiently reliable for the intended analysis.
For applications involving scholarly evidence, measuring hallucination rates may be relevant, but the operational definition of hallucination must be explicit.
Should researchers repeat AI tests?
Repeated testing may be necessary when generative AI outputs vary across executions.
A model that answers a question correctly once may produce an incorrect answer when the same prompt is repeated.
Researchers should determine whether the study evaluates single-response performance, average performance across repeated generations, or a defined aggregation procedure.
The number of repetitions should be justified by the research objective, expected variability, and desired precision.
Repeated generations should not automatically be treated as independent evaluation items.
When relevant, response stability should be evaluated alongside performance.
How can researchers compare AI models fairly?
Comparative studies should use common evaluation objectives and appropriately comparable testing conditions.
Models should generally receive the same underlying test items, although identical prompt wording is not always required when systems have different input conventions.
Researchers should document differences in external tools, retrieved information, generation settings, and prompt-development procedures.
When models are evaluated on the same items, the paired structure of the data should be considered in statistical analysis.
The broader principles of fair AI model comparison are particularly important when the study intends to identify a superior system.
What role should human evaluators play?
Human evaluation may be necessary when the task involves nuanced judgments that cannot be adequately assessed through exact matching or automated tests.
Evaluators should use clear criteria and receive sufficient guidance to apply them consistently.
Where feasible, model identity can be concealed to reduce expectation-related bias.
Researchers should also consider inter-rater agreement and procedures for resolving disagreements.
Human judgment is not automatically an infallible reference standard. Evaluators may disagree, misunderstand the task, or favor particular response styles.
Automated AI-based grading can reduce some practical burdens but introduces its own validation requirements.
How should statistical uncertainty be reported?
Performance estimates are subject to uncertainty arising from the selected evaluation items, repeated generations, measurement procedures, and other sources.
Researchers should report confidence intervals or other appropriate uncertainty measures when making inferential claims.
For model comparisons, effect sizes and paired differences are often more informative than separate percentages alone.
The statistical method should reflect the structure of the data, including clustering by evaluation item and repeated observations where applicable.
Researchers should distinguish statistical significance from practical importance.
A small accuracy advantage may not justify substantially greater cost, slower performance, or increased oversight requirements.
Should cost, speed, and human oversight be included?
For practical deployment studies, performance may include more than the quality of the generated answer.
An AI system that produces highly accurate responses but requires extensive human correction may offer limited efficiency gains.
Likewise, a slightly less accurate model may be more suitable for a low-risk application if it is substantially faster or less expensive, provided the remaining errors are acceptable.
Researchers should define these trade-offs according to the intended use rather than treating them as universal preferences.
When relevant, evaluation may include latency, financial cost, computational requirements, and the amount of human review needed to achieve acceptable outcomes.
These measures should be reported separately unless a composite measure has a defensible rationale.
How should subgroup performance be evaluated?
Average performance may conceal important differences across populations, languages, document types, or task categories.
For example, an AI model may perform well on English-language research abstracts but less reliably on abstracts written in another language.
Researchers should identify subgroups relevant to the intended application and assess whether the evaluation dataset provides adequate coverage.
Subgroup estimates may be imprecise when sample sizes are small. Researchers should report uncertainty and avoid interpreting exploratory differences as established disparities without sufficient evidence.
Subgroup analysis is especially consequential when AI outputs could affect access, opportunity, or high-stakes decisions.
How should AI evaluation findings be reported?
A transparent report should describe the evaluated system, task, dataset, reference standard, prompts, settings, testing period, metrics, statistical analysis, and relevant limitations.
Researchers should preserve unsuccessful outputs and avoid reporting only favorable examples.
Where feasible and permitted, supplementary materials or repositories can provide evaluation data, scoring rubrics, prompts, and analysis code.
Conclusions should identify the conditions under which the observed performance was obtained.
A finding that a model achieved high accuracy on one benchmark does not establish that it is reliable across every task or deployment environment.
