Should AI Be Allowed Retrieval, Tools, or External Sources During Human-vs.-AI Comparisons?

Is It Fair for AI to Use External Tools When Being Compared With Humans?

You are comparing a generative AI system with human researchers on their ability to identify appropriate statistical methods. The AI system can search methodological documentation, execute Python code, and retrieve information from external sources.

Human participants, however, must answer using only the information presented in the evaluation questions.

The AI performs better. Does this demonstrate superior methodological reasoning, or does it partly reflect access to additional information and computational resources?

The distinction matters because contemporary AI applications increasingly combine language models with retrieval systems, code execution, databases, and other external tools. These capabilities can improve performance, but they also change what the evaluation measures.

Researchers need to determine whether the objective is to evaluate the underlying model, the complete AI system, or its practical performance relative to human workflows.

AI Tool Access Should Match the Intended Evaluation

Evaluating Tool-Augmented AI Without Confusing System Performance With Model Capability

What counts as an external tool in AI evaluation?

An external tool is a resource or computational capability used by an AI system beyond generating responses solely from the information supplied in its immediate model context.

Depending on the architecture, external capabilities may include web search, document retrieval, database queries, code execution, mathematical computation, file analysis, and specialized software interfaces.

Researchers should distinguish these capabilities from ordinary prompt content.

For example, supplying an article directly within a prompt differs operationally from allowing an AI system to search a database and select which articles to retrieve.

The second procedure introduces additional components whose behavior may influence the final result.

What is the difference between model-level and system-level evaluation?

A model-level evaluation may be useful when researchers want to examine performance without external retrieval.

A system-level evaluation may be more appropriate when the research concerns whether an AI application can complete real-world tasks.

Researchers should not attribute every improvement in a tool-augmented system exclusively to the language model.

Which external tools can influence AI performance?

The evaluation should document which tools were enabled and how their outputs contributed to the task.

When should AI tools be disabled?

Disabling tools may be appropriate when researchers want to isolate the model's performance under restricted conditions.

For example, a study evaluating unaided mathematical problem solving may prohibit code execution.

Similarly, an evaluation of factual recall may restrict web retrieval so that the model cannot obtain answers from external sources during testing.

These restrictions can help distinguish performance under different information-access conditions.

However, disabling tools may reduce the ecological validity of an evaluation if the intended application ordinarily uses them.

Researchers should explain why the restricted condition is relevant rather than treating tool-free evaluation as the only legitimate benchmark.

When should AI tools be enabled?

Tool access is appropriate when the research concerns a complete AI application that ordinarily relies on external resources.

For example, an AI research assistant designed to retrieve and summarize journal articles should reasonably be evaluated with its retrieval capabilities enabled.

Disabling retrieval would change the system being evaluated.

Likewise, an AI application intended to analyze datasets through code execution may require access to its computational environment.

However, researchers should describe the evaluated system accurately and avoid claiming that its performance represents the unaided capabilities of the underlying model.

What is retrieval-augmented generation?

Retrieval-augmented generation, commonly abbreviated as RAG, combines a language model with a process that retrieves external information for use during generation.

In a research application, the retrieval component might identify relevant passages from journal articles, institutional documents, or methodological references.

The model then generates an answer using the retrieved information.

RAG can improve access to relevant evidence, but it does not guarantee factual correctness.

The system may retrieve irrelevant material, overlook important documents, misinterpret retrieved passages, or generate claims unsupported by the available evidence.

Researchers should therefore evaluate retrieval quality and answer quality separately when the research question requires it.

How can retrieval quality affect human-AI comparisons?

Retrieval determines which information becomes available to the model.

If the retrieval component consistently identifies authoritative sources, the AI system may perform well even when the underlying model has limited internal knowledge of the topic.

Conversely, poor retrieval may produce weak answers despite the model's ability to interpret relevant evidence.

Researchers should distinguish retrieval failures from generation failures.

For example, an incorrect answer may result because the system failed to retrieve the necessary article or because the model misinterpreted an article that was successfully retrieved.

These failure modes have different implications for system improvement.

Should humans receive access to the same external sources?

When the study aims to compare performance using equivalent evidence, access to the same relevant source material may strengthen validity.

However, identical information access does not necessarily require identical interfaces.

A human researcher may use a database search interface while the AI system uses an automated retrieval API.

Researchers should determine whether the study compares information-processing capability or complete workflows that include information retrieval.

If only the AI receives additional evidence, the study should acknowledge that the performance difference may partly reflect unequal information access.

The principles of human tool access in AI comparisons apply to the design of these conditions.

How can researchers separate retrieval benefits from model capability?

An ablation design can compare the same AI model with and without a particular external capability.

For example, researchers might evaluate a language model under a no-retrieval condition and a retrieval-enabled condition.

Both conditions can use the same evaluation items and scoring procedures.

The observed difference provides evidence about the contribution of retrieval under the tested configuration.

However, causal interpretation requires attention to other changes introduced by the experimental conditions.

If enabling retrieval also changes prompts, context length, or generation settings, the observed difference may not be attributable solely to retrieval.

How can an ablation study be structured?

A simple design may compare three conditions: the language model without retrieval, the same model with retrieval, and a relevant human comparator.

Researchers may add a human-AI collaboration condition when the intended application involves assistance rather than replacement.

The analysis should identify which comparisons are primary and which are exploratory.

Should AI be allowed code execution?

Code execution may be appropriate when the evaluated application is designed to perform computational tasks.

For example, a system intended to analyze research datasets may use Python or R to calculate statistics.

However, this changes the interpretation of performance.

A correct numerical result obtained through executed code does not necessarily establish that the language model could perform the calculation accurately without external computation.

Researchers should distinguish code-generation quality, code-execution success, and the correctness of the resulting analysis.

When comparing AI with human researchers, the human condition should also reflect the intended access to statistical software.

Should AI have access to current information?

Current information may be necessary for tasks involving recent publications, updated guidelines, or changing scientific evidence.

A model evaluated without retrieval may be unable to answer questions about developments occurring after its training period.

However, allowing live web access introduces variability because search results, webpages, and indexed information can change.

Researchers should record testing dates and preserve relevant retrieved material where permitted.

For reproducibility, a fixed document corpus may be preferable when the research question does not require live information.

The choice should reflect the intended application rather than automatically favoring either static or live retrieval.

Can external tools create benchmark contamination?

Yes. An AI system with unrestricted retrieval may locate benchmark questions, answer keys, or worked solutions online.

High performance under these conditions may reflect access to existing answers rather than the capability the benchmark was intended to measure.

Researchers should examine whether external sources contain evaluation items or reference answers.

Where necessary, retrieval can be restricted to a controlled corpus that excludes answer keys.

However, contamination risks may also arise from model training data, which researchers may be unable to inspect fully.

These limitations should be acknowledged rather than assumed absent.

How should tool calls be documented?

Tool-augmented AI evaluations may require records beyond the initial prompt and final response.

Depending on the system, useful records include tool names, versions, parameters, queries, execution timestamps, returned information, and relevant errors.

For retrieval systems, researchers may preserve document identifiers, retrieval scores where available, and the passages supplied to the model.

For code execution, relevant scripts, software dependencies, and execution outputs may be necessary.

Researchers should preserve records subject to confidentiality, licensing, and data-protection requirements.

These materials can support reproducibility of AI research involving changing systems.

How should tool failures be evaluated?

External tools can fail independently of the language model.

Search services may return no results, databases may become unavailable, and code execution may encounter errors.

Researchers should define whether failed tool calls count as task failures, whether retries are permitted, and how incomplete executions are handled.

Excluding failed cases without justification may overestimate system performance.

For deployment-oriented studies, reliability should reflect the complete system, including the consequences of tool failures.

Researchers may report model errors and tool-related failures separately while retaining an overall task-success measure.

How should cost and speed be measured when tools are enabled?

External tools can introduce additional processing time and expense.

A retrieval-augmented system may require multiple searches, document processing, and additional generation steps.

Researchers should distinguish model inference time from total system completion time.

Similarly, cost analysis may need to include retrieval services, database access, code execution, and other external operations.

The broader principles of evaluating AI cost and speed are relevant when comparing tool-enabled systems.

A tool-augmented model may achieve higher accuracy while requiring greater resources.

Should researchers measure human oversight for tool-enabled AI?

Yes, when human review is part of the intended workflow.

Tool access can reduce some errors while introducing others.

For example, a retrieval-enabled system may provide genuine references but attach them to claims they do not support.

Researchers may still need to verify source relevance, interpret retrieved evidence, or inspect generated code.

Consequently, system-level performance should consider the effort required to obtain an acceptable final result.

The amount of human oversight required may remain substantial even when external tools improve raw task accuracy.

How should tool-enabled AI performance be reported?

A transparent report should identify the underlying model, enabled tools, relevant configurations, external information sources, and testing period.

Researchers should specify whether results represent model-only or complete-system performance.

Where possible, tool calls and relevant outputs should be preserved for independent examination.

Comparative findings should distinguish differences associated with information access, computational assistance, and task execution.

The conclusion should identify the evaluated configuration rather than attributing every observed advantage to the language model alone.

Comparing AI With and Without Retrieval Against Human Researchers

The hypothetical results suggest that retrieval may improve performance under the tested configuration.

However, the observed advantage does not establish that retrieval alone caused the improvement unless other relevant conditions were appropriately controlled.

The comparison also does not establish that the AI system is equivalent to human experts merely because its score approaches the human average.

Common Misconceptions About AI Tool Access

Deciding Whether AI Should Use Retrieval or External Tools

The appropriate decision depends on whether the study investigates model capability or the performance of a complete AI system.

Researchers should avoid describing a tool-enabled AI system as though it were operating solely from internal knowledge.

Likewise, disabling tools should not automatically be treated as a more rigorous design when the intended application depends on those capabilities.

Before Enabling External Tools in an AI Comparison Study

Frequently Asked Questions About AI Tool Access in Human Comparisons

Evaluate the AI System That the Research Question Actually Concerns

How to Cite This Guide

This guide is intended to be read, shared, and used in research, teaching, and academic work. If you draw on its ideas, explanations, or other content, please acknowledge the source by citing the guide. Doing so gives appropriate credit and helps your readers locate the original resource.