Should Researchers Document AI System Instructions When They Are Available?

Should You Record the Instructions Governing an AI System You Study?

You submit the same research prompt to two AI systems. Both use comparable language models, yet their answers differ substantially. One refuses to complete the task, while the other provides a detailed response. Is the difference attributable to model capability, or could instructions operating behind the interface explain it?

Researchers studying generative AI often document their prompts and model names while overlooking the system-level instructions that may shape responses before the user's prompt is processed.

This raises a methodological question: when system instructions are available, should they become part of the research record, and what should researchers do when those instructions are inaccessible?

Document Available System Instructions When They May Affect the Study

Why AI System Instructions Matter for Research Methodology

What are AI system instructions?

System instructions are directives provided to an AI model or application to establish aspects of its expected behavior. Depending on the platform, they may specify the model's role, response format, operating constraints, or how certain categories of requests should be handled.

In systems that support multiple instruction levels, system messages may have higher priority than ordinary user messages. Some platforms also support developer instructions, application-specific configurations, and additional contextual material.

These mechanisms are not identical across AI providers. The terminology, implementation, and instruction hierarchy may differ.

A developer instruction may occupy a separate priority level between system and user instructions. Researchers should therefore avoid treating every instruction outside the visible user prompt as technically equivalent.

How can system instructions influence research results?

Instructions may influence how a model interprets a task, what information it includes, and which responses it considers appropriate.

For example, a system instruction might direct an AI assistant to answer questions as a mathematics tutor, avoid providing complete solutions, or return responses in a particular structure.

If researchers evaluate the system's ability to solve mathematics problems, those instructions could materially influence the observed performance.

Instructions may also affect refusal behavior, response length, the presentation of uncertainty, or the extent to which a model follows task-specific directions.

Consequently, two researchers using the same model and visible user prompt may not be testing equivalent conditions if their applications provide different higher-priority instructions.

This does not mean that every difference between AI outputs can be attributed to system instructions. Model updates, sampling configurations, conversation history, retrieval tools, and other implementation details may also contribute.

Which instructions should researchers document?

The appropriate documentation depends on the research question and the degree of control available to the researcher.

Instructions should be preserved in their original form when exact wording could affect interpretation. A paraphrase may be useful in a published methods section, but it should not replace the original text in the underlying research record when the original is available and can lawfully be retained.

What if the AI provider does not reveal its system instructions?

Many commercial AI interfaces do not expose every instruction governing their behavior. Researchers may be unable to inspect provider-controlled system messages, safety configurations, or other internal mechanisms.

This limitation should be reported explicitly.

A suitable methodological statement might explain that the researchers controlled the visible user prompts but did not have access to the provider's internal system instructions. The findings would therefore describe the behavior of the accessible AI service under the tested conditions rather than isolate the behavior of an unconstrained underlying model.

Researchers should not claim that a system had no hidden instructions merely because none appeared in the interface.

Nor should they treat a model's self-reported description of its internal instructions as independently verified evidence. AI-generated explanations about hidden configurations may be incomplete or inaccurate.

Should system instructions be included in the published paper?

When instructions are central to the experimental manipulation, their exact wording should generally be reported in the manuscript, supplementary materials, or an appropriately accessible research repository.

For studies where instructions are peripheral, a concise description may be sufficient, provided the underlying record preserves the necessary detail.

The reporting decision should consider whether another researcher could understand the tested condition, assess the interpretation, and reproduce the accessible elements of the procedure.

Documentation may also be constrained by confidentiality agreements, platform terms, institutional security requirements, or restrictions on disclosing proprietary information. Researchers should not publish protected instructions without authorization.

Can system instructions change without researchers knowing?

Yes. Commercial AI providers may revise application behavior or internal configurations without exposing every change to users.

Even when a publicly visible model identifier remains unchanged, researchers should not assume that all surrounding application instructions and service configurations remain constant.

This creates an important distinction between reproducing the original research procedure and obtaining identical model outputs.

Recording the accessible instructions, service identifier, configuration, and testing period improves methodological transparency. It does not guarantee identical future responses.

Researchers should therefore record the AI model version being studied and document when the AI system was tested.

How do system instructions relate to the broader prompt record?

System instructions should not be documented in isolation when other contextual information materially influences the response.

Depending on the study, relevant records may include the user prompt, developer instructions, previous conversation turns, model settings, available tools, and retrieved documents.

A complete record should preserve the distinction between these elements rather than combine them into one undifferentiated transcript.

This becomes especially important when preserving prompts used to study an AI model or evaluating how conversation history affects generative AI responses.

Testing How System Instructions Affect AI Feedback

Without the exact system instructions, another researcher would know that two conditions existed but could not reliably reconstruct the experimental manipulation.

The instructions are therefore part of the method, not merely background information.

Common Mistakes When Reporting AI System Instructions

Deciding How Much System-Instruction Documentation Your Study Needs

The extent of documentation should reflect the methodological role of the instructions.

For studies comparing AI models, instruction transparency also affects the fairness of the comparison. Two systems receiving the same visible prompt may still operate under different application-level conditions.

Researchers planning fair comparisons between AI models should identify which instruction layers they controlled and which remained outside their observation.

What to Record About AI System Instructions

Frequently Asked Questions About Documenting AI System Instructions

Document the Instructions You Can Observe and the Limits You Cannot Remove

How to Cite This Guide

This guide is intended to be read, shared, and used in research, teaching, and academic work. If you draw on its ideas, explanations, or other content, please acknowledge the source by citing the guide. Doing so gives appropriate credit and helps your readers locate the original resource.