Should Large-Scale AI Research Report Computational Resource Use?

Can Other Researchers Evaluate an AI Study Without Knowing What Computing Resources It Used?

A research paper describes an experiment comparing several language models across hundreds of thousands of documents. The authors explain their evaluation metrics and report model performance, but they provide little information about the computing infrastructure used to conduct the experiments.

Readers cannot determine how long processing took, which accelerators were required, whether models were quantized, or how much computation was involved in repeated experiments.

Does this omission matter for scientific reproducibility? Should computational resource reporting be treated as an important part of methodological transparency, particularly when the required infrastructure may be inaccessible to other researchers?

Yes, Material Computational Requirements Should Be Reported

What Large-Scale AI Studies Should Disclose About Computing Resources

Why computational resource reporting is part of methodological transparency

Research methods should provide sufficient information for readers to understand how results were produced and evaluate whether the procedures are scientifically defensible.

In computationally intensive AI research, hardware and execution conditions may influence processing speed, numerical behavior, experimental feasibility, and the practical ability to reproduce results.

For example, a model evaluated using specialized accelerators may not be feasible on ordinary research workstations. A large-scale experiment may also involve extensive preliminary testing that is not apparent from its final reported results.

Computational resource reporting helps readers distinguish methodological choices from infrastructure constraints.

The National Academies' report Reproducibility and Replicability in Science emphasizes the importance of transparent methods, computational artifacts, and appropriate documentation for evaluating research findings.

Resource disclosure does not independently establish reproducibility, but it can make the conditions of reproduction more understandable.

What counts as computational resource use?

Computational resource use refers to the computing capacity and associated resources consumed or allocated while conducting a computational activity.

For AI research, this may involve processors, accelerators, memory, storage, networking, and execution time.

The relevant measurements depend on whether researchers are training models, fine-tuning existing models, or conducting inference.

Researchers should not assume that every metric is necessary for every study. The objective is to identify which resources are material to understanding the reported computational work.

Why hardware specifications matter

Different processors and accelerators can have substantially different computational capabilities, memory capacities, and power requirements.

Reporting only that an experiment used a GPU provides limited information about the actual environment.

Researchers should identify relevant hardware models, accelerator counts, memory capacity, and system configuration when these characteristics affect execution.

For distributed experiments, the number of computing nodes and the arrangement of accelerators may also be important.

However, hardware descriptions should remain proportionate. Reporting every peripheral device may add little value when the scientific procedure depends primarily on the accelerator and inference software.

How should GPU-hours be calculated?

GPU-hours describe the duration for which a specified number of GPUs are allocated to a workload.

GPU-hours are useful for describing allocated capacity, but they do not directly measure actual utilization, floating-point operations, or electricity consumption.

Forty-eight GPU-hours on one accelerator model may represent a different computational capability from forty-eight GPU-hours on another.

Researchers should therefore report the hardware model alongside the allocation and avoid presenting GPU-hours as a universal measure of computational work.

What is the difference between wall-clock time and accelerator-hours?

Wall-clock time measures elapsed duration. Accelerator-hours account for the number of allocated devices over that duration.

A computation running for ten hours on one GPU has a wall-clock duration of ten hours and an allocation of ten GPU-hours.

The same ten-hour execution using eight GPUs represents eighty GPU-hours.

These measures answer different questions. Wall-clock time helps assess how long an experiment takes, while accelerator-hours describe allocated computing capacity.

Neither establishes how efficiently the devices were used.

Researchers should avoid reporting one as though it were equivalent to the other.

Should AI studies report FLOPs?

Floating-point operations, commonly abbreviated FLOPs, describe numerical arithmetic operations performed during computation.

Estimated FLOPs can help characterize computational demand, especially when comparing training procedures or model architectures.

However, FLOP estimates depend on assumptions about the operations counted, model architecture, sequence lengths, and implementation.

A theoretical operation count is not necessarily equivalent to a hardware-measured operation count.

Researchers should explain the estimation method and avoid presenting theoretical calculations as direct measurements.

FLOPs also do not independently determine electricity consumption. Hardware efficiency, memory movement, and system utilization affect energy use.

Why model configuration must accompany hardware reporting

Two studies may use the same GPU model but execute different computational workloads because their model configurations differ.

Relevant details can include model architecture, checkpoint, parameter count, numerical precision, quantization, batch size, context length, and inference implementation.

For training experiments, optimizer settings, training steps, sequence lengths, and distributed-training configurations may be relevant.

For inference studies, decoding parameters and output limits can affect computational demand.

Researchers should report the configuration actually used rather than relying on a general model-family name.

This is especially important when hardware requirements for local AI models are influenced by numerical precision and context length.

How should training and inference resources be distinguished?

Training and inference are different computational activities.

Training modifies model parameters using data and optimization procedures. Inference uses an existing model to process inputs and generate outputs.

Fine-tuning represents an additional training process and may have resource requirements distinct from both initial training and ordinary inference.

Researchers should report these activities separately when their resource demands are material.

For example, a study may require substantial GPU-hours to fine-tune a model but comparatively modest resources to evaluate it on a test dataset.

Combining both activities into one unexplained computational total can obscure the resources needed to reproduce different stages of the work.

Should unsuccessful experiments and preliminary testing be included?

The answer depends on the reporting objective.

If the goal is to estimate the resources required to execute the finalized analysis, the final execution workload may be the appropriate boundary.

If the goal is to describe the total computational expenditure required to develop and evaluate the method, preliminary experiments, unsuccessful runs, and validation should also be considered.

Researchers should distinguish these quantities rather than treating them as interchangeable.

When complete historical usage records are unavailable, researchers should state the limitation rather than inventing a comprehensive total.

The related question of reporting computational costs of AI-assisted research also requires distinguishing resource consumption from monetary expenditure.

How should cloud API workloads be documented?

Researchers using hosted AI services may not have access to the underlying GPUs, memory allocation, or electricity measurements.

They may instead have reliable information about the model identifier, number of requests, input and output tokens, processing dates, and billed usage categories.

These measurements can characterize the workload without revealing the provider's internal computational infrastructure.

Researchers should report what is known and distinguish it from unavailable information.

For example, a study can accurately report processing 30 million input tokens without claiming to know the precise number of GPU-hours consumed by the provider.

Token counts and API charges should not be converted into GPU-hours or electricity consumption using unsupported universal factors.

Should computational energy consumption also be reported?

When reliable measurements are available and energy consumption is relevant to the study, reporting it can improve understanding of resource requirements.

Researchers should identify whether the measurement covers accelerators, an entire workstation, or a broader facility.

Energy consumption is different from computational allocation. Two experiments with the same GPU-hours may consume different amounts of electricity.

Energy reporting should also be distinguished from greenhouse-gas emissions, which depend on electricity sourcing and accounting assumptions.

The role of energy consumption in choosing AI research methods becomes especially relevant when alternative approaches provide comparable scientific performance.

Can computational resource reporting improve reproducibility?

Yes, although its contribution is limited.

Resource information helps other researchers assess whether they have sufficient infrastructure to repeat a workflow and identify execution conditions that may affect results.

However, knowing that an experiment used a particular GPU for a specified duration does not provide the model weights, training data, prompts, preprocessing code, or evaluation procedures needed to reproduce the analysis.

Computational resource reporting should therefore complement methodological and artifact documentation.

The National Academies' discussion of reproducibility and the ACM artifact review framework both support the importance of accessible and adequately documented computational procedures.

For studies using open-weight models to improve research reproducibility, preserving model checkpoints and software configurations remains necessary even when resource use is fully reported.

Why resource reporting matters for research equity

Computationally intensive AI research may depend on resources unavailable to some institutions.

When authors report hardware requirements and processing demands, other researchers can better assess the practical feasibility of repeating or extending the work.

This transparency does not eliminate unequal access, but it makes resource barriers more visible.

A model that requires specialized accelerators may be technically reproducible yet financially inaccessible to many research teams.

The concern connects to institutional inequalities in access to AI resources.

Transparent reporting can also support the development of lower-resource alternatives and more realistic computational budgets.

What reporting standards should researchers consult?

Researchers should examine the requirements of their target journal, conference, funder, and institution.

The NeurIPS Paper Checklist provides an established example of research reporting expectations in machine learning, including computational experiments and reproducibility-related information.

The ACM artifact review and badging framework offers additional guidance on making computational artifacts available and assessable.

However, requirements vary across disciplines and venues. A clinical study using AI for a minor supporting task may not require the same computational disclosure as a paper introducing a large language model.

Researchers should apply proportionality while ensuring that materially important information is not omitted.

How should computational resources be presented in a manuscript?

For substantial AI experiments, a concise table can provide readers with the most relevant information without overloading the main narrative.

The methods section should explain the computational environment, while supplementary materials may provide detailed configurations and usage records.

Useful fields can include model identifier, task, hardware, device count, processing duration, allocated accelerator-hours, precision, workload size, and the measurement or estimation basis.

Researchers should distinguish planned allocations from observed consumption and identify unavailable measurements.

Clear reporting is preferable to an extensive collection of figures whose meanings and boundaries are unspecified.

Reporting Resources for a Large-Scale AI Document Classification Experiment

The total allocated GPU time is 20 + 40 + 48 + 60 = 168 GPU-hours.

The 57 stage-hours represent the sum of elapsed durations for the four listed stages. This is not necessarily the total calendar time of the project, particularly if stages overlap.

Model A uses 60 GPU-hours across its reported stages, while Model B uses 108 GPU-hours.

These figures do not establish that Model B consumes 80% more electricity. Such a conclusion would require appropriate power measurements or defensible energy estimates.

In an actual manuscript, the authors should also identify the GPU models, model checkpoints, software versions, numerical precision, and relevant processing settings.

If they have reliable power measurements, electricity consumption can be reported separately with its measurement boundary.

The example demonstrates how resource reporting can be transparent without claiming to measure quantities that were not observed.

Common Mistakes in Computational Resource Reporting

How to Establish a Defensible Computational Reporting Procedure

Researchers should begin recording computational resources when experiments start rather than reconstructing them after analysis.

A simple resource log can identify each experimental stage, model, hardware configuration, processing duration, workload volume, and relevant usage measurements.

For cloud services, record provider-reported usage information and identify which infrastructure details are unavailable.

For publication, use a concise resource table supported by sufficient methodological detail to explain the measurements.

Computational transparency should help readers assess what the experiment required and what would be needed to reproduce it, without implying that greater resource consumption demonstrates greater scientific merit.

Before Reporting Computational Resources in AI Research

Questions About Computational Resource Reporting in AI Research

Computational Transparency Makes Large-Scale AI Research More Assessable

How to Cite This Guide

This guide is intended to be read, shared, and used in research, teaching, and academic work. If you draw on its ideas, explanations, or other content, please acknowledge the source by citing the guide. Doing so gives appropriate credit and helps your readers locate the original resource.