03 · What You Need to Know
Measuring AI Efficiency Without Sacrificing Evaluation Validity
Why should cost and speed be considered in AI evaluation?
AI systems consume resources to perform tasks. These may include computing infrastructure, paid API access, electricity, network services, and human labor.
For a small experimental evaluation, these requirements may be relatively modest. For an institutional application processing thousands of documents, small differences in per-request cost or processing time can accumulate substantially.
Consider an educational research team screening 20,000 abstracts. A model that costs only a few cents more per item may create a meaningful difference in total expenditure.
However, cost alone cannot establish value. A less expensive system that misses relevant studies may introduce additional work or undermine the research objective.
Researchers should therefore interpret efficiency in relation to the quality and consequences of the resulting outputs.
What is the difference between latency and throughput?
Latency and throughput describe different aspects of processing speed.
Latency
The elapsed time required to complete a defined request or operation, measured from a specified starting point to a specified endpoint.
Throughput
The amount of work completed per unit of time, such as processed abstracts per minute or completed requests per second.
A system may have relatively high latency for individual requests while achieving substantial throughput through parallel processing.
Conversely, a model may respond quickly to one request but perform poorly when many requests are submitted concurrently.
Researchers should identify whether the intended application prioritizes rapid individual responses, high-volume processing, or both.
How should response latency be measured?
Latency measurement requires clearly defined start and end points.
For an API-based evaluation, researchers might measure the time between submitting a request and receiving the complete response.
For a streaming interface, time to first token and time to complete response may provide different information.
Researchers should also distinguish server-side processing time from end-to-end latency, which may include network delays and application overhead.
Timing should be collected across repeated requests rather than inferred from one demonstration.
Median latency and relevant percentiles, such as the 95th percentile, may be useful because response times can be unevenly distributed.
The evaluation should document concurrency, request size, output length, and other conditions affecting the measurement.
What is time to first token?
Time to first token refers to the elapsed time between a defined request starting point and the arrival of the first generated token or streamed output unit.
It can be relevant for interactive applications because users may perceive a system as responsive when output begins appearing quickly.
However, time to first token does not indicate how long the complete task takes.
A model may begin responding almost immediately but require considerable additional time to finish a long explanation.
For research workflows requiring complete outputs before processing can continue, total completion time may be more relevant.
Researchers should avoid using the first visible output as a substitute for full-response latency without explaining that choice.
How should throughput be calculated?
Throughput measures completed work relative to elapsed time.
This measure describes processing volume under the tested conditions.
It does not establish that all 600 classifications were correct or that the system could maintain the same throughput under different workloads.
Researchers should report whether failed requests, retries, and rate-limit delays were included.
What costs should researchers include?
The appropriate cost boundary depends on the evaluation objective.
An API benchmark may focus on the provider's charges for model usage. A deployment-oriented study may need to include additional operational expenses.
| Cost component |
Possible measurement |
Research consideration |
| API usage |
Charges for input, output, or other billable units |
Record the applicable pricing and usage period |
| Computing infrastructure |
Hardware rental, processing time, or allocated infrastructure expenses |
Relevant for self-hosted and hybrid systems |
| External tools |
Retrieval, search, storage, or code-execution charges |
May contribute substantially to total workflow cost |
| Human verification |
Reviewer time and associated labor cost |
Important when outputs require checking or correction |
| Retries and failures |
Additional requests and processing resources |
May increase effective cost per usable output |
| Implementation and maintenance |
Development, integration, monitoring, and upkeep |
Relevant for deployment or economic evaluations |
Researchers should not combine these costs indiscriminately. A narrow inference-cost study and a comprehensive deployment-cost study answer different questions.
The report should identify which expenses were included, excluded, estimated, or unavailable.
Why is token consumption relevant?
Many language model services charge according to token usage or related billing units.
Input tokens, generated output tokens, cached inputs, and other processing categories may be priced differently depending on the provider and service.
Researchers should record the relevant usage information and the pricing rules applicable during testing.
Token counts should not automatically be treated as equivalent measures of information across different tokenizers or languages.
A task expressed in one language may require a different number of tokens than an equivalent task expressed in another.
Consequently, token efficiency should be interpreted in relation to the task and model-specific tokenization procedures.
How can researchers calculate cost per correct answer?
One descriptive measure divides the total cost of a defined evaluation workload by the number of correct outputs produced.
This measure can be useful for comparing resource efficiency under a common evaluation workload.
However, it does not capture the consequences of incorrect outputs, human correction requirements, or differences in task difficulty.
Researchers should avoid interpreting a lower cost per correct output as automatic evidence of practical superiority.
Should researchers measure cost per usable output instead?
For some applications, cost per usable output may be more meaningful than cost per technically correct response.
A generated research summary might be largely accurate but still require substantial editing before it can be used.
Researchers can define usability through explicit acceptance criteria, such as satisfactory factual accuracy, completeness, and formatting compliance.
The total cost can then be divided by the number of outputs meeting those criteria.
However, usability thresholds should be justified and established before final evaluation.
Otherwise, the researcher may inadvertently define usability in a way that favors a preferred system.
Why should human correction time be included?
AI systems can produce outputs rapidly while shifting substantial work to human reviewers.
For example, a model may summarize an article in seconds, but a researcher may need several minutes to verify citations, correct interpretations, and restore omitted methodological details.
If the intended workflow requires such review, measuring only generation time may exaggerate the apparent efficiency gain.
Researchers should distinguish automated processing time from total task completion time.
They should also record the amount of correction or verification required to reach an acceptable output.
The role of human oversight in AI performance evaluation is particularly relevant when assessing real-world productivity.
How should AI speed be compared with human performance?
Comparisons between AI and humans should reflect the research objective.
If the study examines unaided task completion, a controlled comparison may be appropriate.
If the study concerns realistic research workflows, human participants should ordinarily have access to the tools and resources that competent practitioners would normally use.
Researchers should avoid comparing the AI's raw generation time with the entire human research process unless the tasks and output requirements are genuinely comparable.
For example, producing a preliminary summary is not equivalent to delivering a verified summary suitable for a systematic review.
Both speed and output acceptability should be considered.
How should researchers compare models with different response lengths?
Response length can influence latency, token usage, and cost.
A model producing a 1,000-word explanation may require more processing than one returning a single classification label.
When the task permits standardized output constraints, researchers can specify comparable response requirements.
However, artificially forcing identical output lengths may be inappropriate when one system requires additional explanation to satisfy the task.
The study should distinguish efficiency differences attributable to the system from those associated with different output requirements.
Researchers should document response lengths and relevant formatting constraints when they materially affect interpretation.
What is the relationship between accuracy, cost, and speed?
These dimensions may involve trade-offs, but the relationship is not necessarily linear.
A slower or more expensive model may provide substantially better performance on difficult tasks. Another model may achieve similar quality at lower cost.
Researchers should examine whether one system offers a favorable balance for the intended application.
One useful approach is to identify Pareto-efficient systems: alternatives for which no other evaluated system performs at least as well on every relevant dimension and strictly better on at least one.
However, Pareto efficiency depends on the dimensions and direction of preference defined in the analysis.
Researchers should avoid collapsing accuracy, speed, and cost into one arbitrary score without justified weights.
Separate reporting often makes trade-offs easier to interpret.
How should pricing changes be handled?
Commercial AI pricing can change over time.
Researchers should record the pricing schedule applicable during the evaluation and identify whether costs reflect actual invoices, published rates, or modeled estimates.
Comparisons using different pricing periods may be misleading if changes are not accounted for.
Where relevant, researchers can conduct sensitivity analyses using alternative plausible pricing assumptions.
However, projected costs should be distinguished from observed expenditure.
A model's measured computational performance may remain informative after a price change, while the economic interpretation may require updating.
How should cost and speed be reported statistically?
Researchers should collect repeated observations when latency or usage varies across requests.
Latency distributions may be skewed, making medians and percentiles useful alongside means.
For model comparisons, common evaluation items and comparable workload conditions can support paired analyses.
Researchers should report variability and uncertainty where appropriate rather than relying on one timing measurement.
Cost estimates should identify the assumed workload, pricing structure, and treatment of failures or retries.
When results are intended to generalize beyond the tested environment, limitations arising from network conditions, hardware, concurrency, or service load should be acknowledged.
When can cost and speed be omitted?
Not every AI study requires economic or latency analysis.
A study focused on prompt sensitivity, reasoning validity, or factual hallucinations may legitimately concentrate on those outcomes.
However, a study claiming that AI improves productivity, reduces institutional costs, or offers a more efficient alternative requires evidence addressing those claims.
Researchers should select metrics according to the intended inference rather than treating every possible performance dimension as mandatory.
The broader principles of AI performance evaluation remain applicable: metrics should be valid for the task, clearly defined, and interpreted within the tested conditions.
