03 · What You Need to Know
How to Report AI Performance Evidence With Sufficient Scientific Precision
Why Is Citing AI Performance Different From Citing a Stable Scientific Fact?
AI performance is conditional on the system being evaluated and the procedures used to evaluate it.
A statement such as "Model X achieves 95% accuracy" appears straightforward, but it leaves several questions unanswered. Accuracy on which task? Using which dataset? Under what scoring rules? Which model version? Was the evaluation conducted independently? Were the results obtained before or after a significant system update?
These details determine what the reported number means.
A performance estimate may be entirely accurate for one benchmark yet provide limited evidence about performance on another research task. Similarly, a result obtained from an earlier model version may not describe a later commercial deployment.
Researchers should therefore treat AI performance figures as context-dependent empirical findings rather than permanent characteristics of a named product.
What Information Should Accompany an AI Performance Claim?
The amount of detail needed depends on the importance of the claim and the reporting context. A brief literature review may require a concise qualification, while a study that relies on AI performance as a central methodological justification may need considerably more documentation.
The following elements are particularly useful.
| Element |
What to Report |
Why It Matters |
| Evaluation source |
Original study, technical report, or authoritative evaluation documentation |
Allows readers to examine the underlying evidence. |
| System identity |
Model name, version, identifier, and relevant deployment configuration |
Identifies the system actually evaluated. |
| Evaluation period |
Testing date or period, when available |
Helps establish the temporal relevance of the evidence. |
| Task and dataset |
Benchmark, sample, population, or research task |
Defines the scope of the performance estimate. |
| Performance metric |
Accuracy, sensitivity, F1 score, calibration measure, or another defined outcome |
Clarifies what was measured. |
| Evaluation conditions |
Prompts, preprocessing, tools, sampling settings, and reference standards when relevant |
Supports interpretation and potential replication. |
| Evidence limitations |
Uncertainty, known biases, restricted applicability, or evaluation constraints |
Prevents unsupported generalization. |
Not every detail belongs in the same sentence. Researchers can provide essential qualifications in the prose and place fuller technical information in a methods section, table, supplementary material, or accompanying research record.
Should Researchers Cite the Original Study or the AI Vendor's Website?
The preferred source depends on the claim being supported.
If a researcher is discussing a performance estimate established in an empirical study, the original study should ordinarily be cited. If the claim concerns a model's documented release date, version identifier, or available API functionality, official provider documentation may be the more appropriate primary source.
Vendor benchmark reports can also be relevant primary evidence for vendor-conducted evaluations. Their methodological limitations and potential conflicts of interest should be considered rather than assumed away.
Researchers should distinguish between evidence about what a vendor reports and evidence independently establishing how a system performs.
Claim about vendor-reported performance
The citation establishes what the developer reported under its evaluation conditions.
Claim about independently observed performance
The citation supports results obtained through an evaluation conducted with meaningful independence from the developer.
These claims may involve the same model but are not interchangeable. The distinction is particularly relevant when comparing independent and vendor evaluations of AI research tools.
How Should Researchers Phrase AI Performance Claims in Academic Writing?
Careful wording identifies the source of the result and avoids implying more generality than the evidence supports.
| Overgeneralized Wording |
More Defensible Wording |
| The AI model is 95% accurate. |
In the reported evaluation, the specified model achieved 95% accuracy on the stated classification task. |
| This AI system outperforms human researchers. |
The evaluated system achieved a higher score than the human comparison group under the study's particular testing conditions. |
| The latest AI models can reliably identify research errors. |
The cited evaluation reported performance on defined error-detection tasks; its applicability to other research errors remains uncertain. |
| AI Model X is the best model for research. |
Model X achieved the highest reported score among the systems compared on the specified evaluation. |
| The tool has been proven reliable. |
The tool demonstrated the reported performance under the evaluation conditions, although further testing may be needed for the intended research context. |
The more defensible formulations do not weaken the evidence. They describe it more accurately.
Researchers should be especially cautious with words such as "proven," "always," "reliably," and "best" when the supporting evidence concerns a limited evaluation.
Why Should the Model Version Be Identified?
Model versions help establish which system produced the reported outputs.
A model family may contain multiple releases with different training procedures, capabilities, or operational characteristics. Commercial providers may also change the underlying implementation of a product while retaining the same user-facing name.
Where available, researchers should report exact model identifiers rather than relying exclusively on broad product labels.
For API-based research, this may include the model identifier supplied in the request or returned by the service. For locally deployed systems, relevant information may include the model release, weight revision, software dependencies, and inference configuration.
Researchers should distinguish a fixed version identifier from an alias that may resolve to different implementations over time.
When exact version information is unavailable, the limitation should be disclosed rather than replaced with an inferred version.
This matters particularly when a named AI product may change its behavior after an update.
Should Researchers Include the Date When an AI Model Was Evaluated?
Yes, when the evaluation date materially affects the interpretation of the finding.
The publication date of a study does not necessarily identify when its AI experiments were conducted. Manuscript preparation, peer review, and publication can occur substantially later than data collection.
A study published after a major model update may still report experiments conducted using an earlier version.
Researchers should distinguish among the evaluation date, model release date, publication date, and date on which online documentation was accessed.
These dates answer different questions.
The evaluation date helps identify when the reported performance was observed. The publication date identifies when the source became publicly available. An access date can help document when a changeable online resource was consulted.
Where available, an archived version of the source or a persistent release identifier may provide stronger traceability than an access date alone.
What if a Study Does Not Report the Exact AI Model Version?
Researchers should not invent missing methodological information.
If a study identifies only a product name, the citation can still be used, but claims about the evaluated system should reflect that limitation.
For example, a researcher may state that the authors evaluated a named commercial AI service during a specified period, while acknowledging that the precise underlying model version was not reported.
The absence of version information may limit replication and complicate comparisons with later evaluations.
Where the missing information is consequential, researchers may consult supplementary materials, repositories, official documentation, or clarification provided by the original authors.
Uncertainty should remain explicit if the information cannot be established.
How Should Researchers Cite AI Benchmark Results?
Benchmark citations should identify the benchmark and the evaluated system, together with the metric and relevant evaluation protocol.
For example, a score obtained through multiple-choice question answering may depend on the prompting method, number of examples included in the prompt, scoring rules, and benchmark version.
Two studies using the same benchmark name may not have implemented identical procedures.
Researchers should therefore inspect the original methods before comparing reported scores.
Where benchmark contamination or repeated optimization may affect interpretation, the limitation should be acknowledged. The existence of a high score does not establish that the system will perform equally well on unfamiliar research tasks.
These concerns are central to interpreting AI benchmark evidence and assessing possible benchmark contamination.
Can Researchers Compare Performance Figures From Different AI Studies?
Yes, but direct numerical comparisons require methodological compatibility.
Suppose one study reports 90% accuracy and another reports 94%. The second figure does not necessarily establish superior model performance if the evaluations used different datasets, task definitions, reference standards, or scoring procedures.
Differences in sampling uncertainty may also matter. A small numerical advantage may not be distinguishable from ordinary evaluation variability.
Researchers should examine whether the studies evaluated comparable systems under sufficiently similar conditions.
When the conditions differ materially, it may be more appropriate to describe the findings separately rather than rank the systems.
A controlled comparison using the same evaluation dataset and protocol can provide stronger comparative evidence, although its conclusions remain limited to the evaluated conditions.
How Should Researchers Handle Performance Claims That May Have Become Outdated?
Researchers should first determine whether the original claim concerns historical performance or current capability.
Historical findings can remain relevant even after the evaluated system is replaced. They may document technological progress, earlier limitations, or the development of evaluation methods.
However, using an earlier performance figure to justify adopting a current AI system requires additional scrutiny.
Researchers should examine subsequent model changes, newer evaluations, and differences between the original testing conditions and the intended application.
Where current applicability is uncertain, the citation should be framed accordingly.
The broader issue is how rapidly AI performance evidence can lose practical relevance without necessarily losing historical scientific value.
Should Researchers Cite AI-Generated Summaries of Performance Studies?
AI-generated summaries should not replace examination of the original evidence when making substantive performance claims.
A generated summary may omit evaluation conditions, confuse performance metrics, misidentify model versions, or present conclusions more strongly than the source supports.
Researchers should verify numerical results, methodological details, and qualifications against the original publication or authoritative documentation.
If the original source cannot be accessed or verified, researchers should avoid presenting the AI-generated description as independently established evidence.
The central principle is straightforward: cite the source that actually supports the claim, not merely the system that supplied a convenient description of it.
How Much Technical Detail Belongs in a Citation Versus the Methods Section?
A bibliographic citation identifies the source. The surrounding text explains why the source supports the claim.
When researchers conduct their own AI evaluations, the methods section should provide sufficient detail for readers to understand what was tested and how the results were obtained.
Relevant information may include model identifiers, prompts, sampling settings, software versions, evaluation datasets, reference standards, human assessment procedures, and statistical analyses.
Where journal requirements permit, extensive prompts, code, and evaluation materials can be placed in supplementary files or repositories, subject to privacy, licensing, and other restrictions.
Research reporting should aim for transparency about the evaluated system and procedures, without implying that every commercial system can be reproduced exactly.
Does the Required Citation Format Change Because the Subject Is AI?
Not necessarily. Established academic citation styles generally remain applicable to journal articles, conference papers, technical reports, and online documentation concerning AI systems.
The appropriate reference format depends on the source type and the journal's or institution's requirements.
The additional challenge lies in scientific reporting: researchers may need to identify model versions, evaluation dates, and testing conditions within the prose or methods section even when those details are not standard elements of the bibliographic reference.
Researchers should not confuse citing a published study about AI with documenting their own use of an AI system. These are related but distinct reporting activities.
