03 · What You Need to Know
Why Prompt Preservation Matters for AI Research Reproducibility and Integrity
Why are prompts considered research materials?
Prompts influence the conditions under which a generative AI system produces responses.
For example, asking a model to identify methodological limitations may produce different results from asking it to act as a journal reviewer evaluating a manuscript for publication.
Both instructions concern research evaluation, but they establish different contexts and expectations.
When the research question concerns AI behavior, prompt wording can become an experimental variable.
Researchers should therefore treat consequential prompts as part of the documented research procedure rather than disposable conversational instructions.
The broader methodological significance of prompting becomes especially clear when different instructions produce different conclusions about the same model.
Which prompts should researchers preserve?
| Prompt category |
Preservation recommendation |
Reason |
| Formal evaluation prompts |
Preserve the complete set |
They define the observations included in the study. |
| Failed formal prompts |
Preserve with outcomes and failure information |
Exclusion may bias reported performance. |
| Exploratory prompt variants |
Preserve consequential versions and selection history |
They explain how the final procedure was developed. |
| System and developer instructions |
Preserve when accessible and relevant |
They may materially influence model behavior. |
| Follow-up prompts |
Preserve when part of the evaluated interaction |
They establish conversation context. |
| Unrelated informal conversations |
Apply relevance and retention requirements |
They may have no methodological role. |
| Confidential prompts |
Preserve securely where authorized |
Auditability must be balanced with privacy and access restrictions. |
Preservation should reflect the role each prompt played in producing or selecting the reported evidence.
Should every prompt in the formal evaluation be archived?
Generally, yes, particularly when prompts are the experimental inputs used to evaluate AI performance.
For example, a study examining citation hallucinations across 200 research questions should preserve the complete set of questions submitted to the model.
Without those prompts, independent researchers may be unable to determine whether the evaluation tasks were ambiguous, unusually difficult, or systematically biased.
Preserving the prompts also allows the original team to revisit individual errors and verify reported classifications.
However, archival access may need to be restricted when the prompts contain confidential information.
Complete internal preservation does not necessarily require unrestricted public distribution.
Should researchers preserve failed prompts?
Yes, when those prompts were part of formal testing or materially influenced the research design.
For example, a researcher evaluating an AI model's ability to generate valid citations should not discard prompts that produced fabricated references while retaining only successful cases.
Such selective preservation could distort the estimated performance.
However, unsuccessful exploratory prompts used before the formal protocol was finalized may be treated differently from prespecified evaluation trials.
Researchers should distinguish development activities from confirmatory testing.
The important requirement is that exclusions and selection decisions remain traceable and are not used to conceal unfavorable findings.
What is prompt cherry-picking?
Prompt cherry-picking occurs when researchers selectively report or analyze prompts based on favorable outcomes while concealing relevant unsuccessful attempts.
For example, a researcher may test 20 different prompts and publish only the one that produces the highest accuracy, presenting the result as though the prompt had been selected independently of performance.
This can exaggerate the apparent effectiveness of the model or prompting strategy.
The problem is particularly serious when the same evaluation dataset is repeatedly used to optimize prompts and then presented as an independent test.
Researchers should document prompt selection and maintain appropriate separation between development and final evaluation.
Preserving prompt history helps identify whether reported performance reflects generalizable capability or optimization to a particular benchmark.
Should exploratory prompts be treated like formal experimental observations?
Not necessarily.
Exploratory prompting may involve testing possible instructions, discovering task limitations, or refining the research procedure.
Such activity can be legitimate, provided that it is not misrepresented as a prespecified confirmatory evaluation.
Researchers should document consequential exploratory decisions, particularly when they affect model selection, prompt wording, task selection, or reported outcomes.
However, not every incidental wording experiment necessarily requires extensive analysis in the final manuscript.
The distinction between exploratory and formal testing should be established clearly.
Where practical, researchers may preserve a broader development archive while reporting a concise methodological account of the decisions that mattered.
Why is preserving the exact wording important?
Small differences in wording can affect how a model interprets a task.
For example, asking an AI system to identify weaknesses in a study differs from asking it to provide constructive feedback without being overly critical.
The second instruction may influence the severity or framing of the response.
Researchers should therefore preserve the actual prompts submitted rather than rely exclusively on paraphrased descriptions.
A manuscript stating that the model was asked to evaluate research quality may be insufficient if the exact instructions materially influenced the scoring or conclusions.
Verbatim preservation supports independent examination of the experimental conditions.
Should system instructions be preserved?
When accessible and relevant, yes.
System and developer instructions may establish behavioral constraints, roles, output requirements, or priorities that influence model responses.
For example, an AI system instructed to provide brief educational explanations may behave differently from one instructed to produce detailed technical critiques.
However, commercial services may not expose all underlying instructions.
Researchers should preserve instructions they supplied or were authorized to access and acknowledge inaccessible configuration details.
They should not claim to have documented hidden instructions that were unavailable.
The distinction between visible prompts and inaccessible system-level controls should remain explicit.
Does conversation history count as part of the prompt?
For conversational AI systems, earlier messages may form part of the effective input context.
A question submitted after several related exchanges may produce a different answer from the same question submitted in a fresh session.
For example, a model that has already received a research proposal may interpret a later question about methodological limitations in relation to that proposal.
Researchers should therefore preserve relevant conversation history when studying multi-turn interactions.
If the design requires independent responses, fresh sessions may reduce contextual carryover.
However, researchers should still document how those sessions were initialized and whether persistent settings or other context could influence them.
Should every follow-up prompt be recorded?
Yes, when follow-up prompts are part of the evaluated procedure.
For example, a study examining whether an AI model corrects its errors after receiving feedback must preserve the initial question, original response, corrective feedback, and subsequent answer.
Without the complete interaction sequence, the observed improvement may be difficult to interpret.
However, unrelated follow-up exchanges that played no role in the research may not require the same treatment.
The preservation boundary should follow the defined research session and analytical purpose.
What if a prompt contains confidential research information?
Confidentiality obligations may restrict both submission to external AI services and subsequent disclosure of the prompt.
For example, a prompt containing unpublished patentable material, identifiable participant information, or restricted institutional data may require special authorization.
Researchers should assess whether such information can legitimately be processed by the selected AI service before submitting it.
Where preservation is permitted, the prompt may need to be stored in an access-controlled archive rather than a public repository.
Researchers should distinguish the need to preserve research records from permission to disclose their contents.
Reproducibility does not automatically override privacy, confidentiality, or intellectual-property obligations.
Can researchers anonymize prompts before sharing them?
Potentially, provided that the process does not create misleading documentation.
For example, researchers may remove identifiable participant details from prompts before releasing supplementary materials.
However, redaction may change the context that influenced the original model response.
Researchers should therefore preserve an authorized original record where appropriate and identify publicly released versions as redacted.
They should explain whether the redactions affect the ability to reproduce the evaluation.
Replacing confidential details with generic placeholders may protect privacy but does not necessarily reproduce the original computational conditions.
Can prompts be stored in a spreadsheet?
Yes. A structured spreadsheet or database may be suitable for preserving prompt records, particularly in smaller studies.
However, researchers should ensure that long prompts, special characters, multiline content, and structured inputs are preserved accurately.
For larger evaluations, machine-readable formats such as JSON Lines may provide advantages because each record can contain prompts, settings, outputs, and metadata.
Researchers should choose a format that supports reliable retrieval and analysis.
The preservation format matters less than the completeness, integrity, and interpretability of the records.
What metadata should accompany preserved prompts?
Prompt text alone may be insufficient to reconstruct the testing conditions.
Researchers should consider preserving a unique prompt identifier, model identifier, timestamp, testing session, trial number, generation settings, and output reference.
Where relevant, the record may also identify prompt version, experimental condition, tool access, and conversation context.
| Field |
Illustrative value |
| prompt_id |
P001 |
| prompt_version |
v1.2 |
| research_phase |
formal_evaluation |
| model_identifier |
Provider-supplied identifier |
| submitted_at |
2026-10-17T10:30:00+08:00 |
| session_id |
S001 |
| trial_number |
1 |
| response_id |
R001 |
| evaluation_status |
included |
These values are illustrative. Actual records should contain the exact prompt content or an unambiguous reference to its preserved location.
How can prompt version control improve reproducibility?
Version control allows researchers to track changes to prompts and identify which formulation was used during each testing period.
For example, a research team may revise a prompt to clarify output formatting while leaving the substantive task unchanged.
Even apparently minor revisions should be documented when they could affect model behavior.
Researchers may use a version-controlled repository or structured change log to preserve the sequence of revisions.
However, a version number alone is insufficient if the corresponding prompt text cannot be recovered.
The archive should link each version to the observations it generated.
Should researchers preserve the order in which prompts were submitted?
Yes, when order may influence model behavior or interpretation.
In multi-turn conversations, earlier prompts directly affect the context available to the model.
In independent sessions, order may still matter if testing conditions change over time.
For example, testing all difficult questions after a provider update could confound task difficulty with deployment period.
Researchers may randomize or counterbalance order where appropriate.
Preserving the actual sequence allows later examination of possible order effects.
Should researchers preserve unsuccessful API requests?
Generally, when those requests are part of formal testing or affect the interpretation of system reliability.
For example, an API request may fail because of a timeout, rate limit, content restriction, or service error.
These outcomes should not automatically be classified as incorrect substantive answers.
Researchers should define how technical failures are distinguished from model-generated errors.
Preserving request metadata and failure categories can support accurate reporting of both performance and operational reliability.
However, sensitive credentials and authentication tokens should never be included in shared research logs.
Can researchers reproduce AI responses using preserved prompts?
Not necessarily.
Preserving prompts improves procedural reproducibility, but model outputs may depend on other conditions.
Commercial models may change, and even fixed models may exhibit nondeterministic behavior.
Researchers should therefore preserve original outputs as well as the instructions that generated them.
Where possible, relevant model identifiers, settings, timestamps, and conversation histories should also be recorded.
The distinction between reproducing an experimental procedure and obtaining identical text should be explained in the study.
Does preserving prompts prevent prompt-selection bias?
It helps make selection decisions auditable, but preservation alone does not eliminate bias.
A researcher may preserve all tested prompts while still selecting the best-performing formulation using the final evaluation dataset.
Such a procedure can overfit the benchmark.
Researchers should establish appropriate development and evaluation stages and document the criteria used to select prompts.
Independent evaluation data may be necessary when making claims about generalizable performance.
Prompt archives support transparency, while research design determines whether the resulting performance estimates are defensible.
Can prompt archives support independent auditing?
Yes. Complete prompt and response records allow reviewers to examine the conditions under which findings were produced.
For example, an auditor may determine whether prompts contained hints that made a supposedly difficult question easier.
They may also identify excluded failures or inconsistencies between the documented procedure and reported results.
However, archives should include enough context to interpret the records accurately.
A collection of prompts without corresponding outputs, timestamps, or evaluation decisions may provide only limited auditability.
Researchers should preserve relationships among the relevant research materials.
How long should prompt records be retained?
Retention periods depend on institutional policies, funder requirements, contractual obligations, disciplinary practices, and applicable law.
Researchers should not assume that AI prompt records are exempt from ordinary research data-management requirements.
Where prompts constitute primary research observations or consequential methodological records, they may fall within the institution's established retention framework.
Confidentiality and data-protection requirements may also influence storage duration and access.
Researchers should consult the applicable policies rather than adopt an arbitrary universal retention period.
Should prompts be published in supplementary materials?
Where feasible and permitted, publishing consequential prompts can improve transparency and enable independent examination.
For example, a study comparing language models on a fixed set of educational questions may provide the complete prompt set as supplementary material.
However, public disclosure may be inappropriate when prompts contain confidential information, copyrighted material subject to restrictions, or sensitive research records.
Researchers may instead provide redacted examples, controlled access, or a documented explanation of the restrictions.
The manuscript should distinguish materials preserved internally from those available publicly.
Can prompt preservation help explain conflicting AI research findings?
Potentially. Two studies may report different performance results because they used different instructions, contextual information, or follow-up procedures.
For example, a model asked to produce concise answers may omit important qualifications that appear when it is asked for detailed explanations.
Preserved prompts allow researchers to examine whether such differences contributed to conflicting findings.
However, prompt variation is only one possible explanation.
Model versions, testing dates, evaluation criteria, and tool access may also influence outcomes.
Comparative interpretation requires examining the complete methodological context.
What if the researcher forgot to preserve early prompts?
The researcher should determine whether the missing prompts materially influenced the final research design or reported conclusions.
Available records, version histories, API logs, or research notes may help reconstruct some of the development process.
However, exact prompt wording should not be invented retrospectively.
If consequential prompts cannot be recovered, the limitation should be reported where relevant.
The researcher may also need to reconsider claims that depend on undocumented prompt selection or optimization.
Transparent acknowledgment is preferable to presenting an incomplete record as though it were comprehensive.
How should prompt preservation be reported in a research paper?
The methods should explain how prompts were developed, selected, submitted, and archived.
Researchers should identify whether the evaluation used fixed prompts, adaptive prompting, multi-turn conversations, or several prompt variants.
They should also describe how unsuccessful attempts and technical failures were handled.
Where prompts were optimized during development, the selection procedure and separation from final evaluation should be documented.
The manuscript should identify which materials are publicly available and which remain restricted.
Prompt preservation is most useful when it allows readers to understand the actual procedure rather than merely confirming that some instructions were saved.
