03 · What You Need to Know
How to Document Major Research Decisions Influenced by AI
Why Does Documenting AI-Influenced Decisions Matter?
Research decisions often develop over time. An analytical method may change after assumptions are examined. A theoretical interpretation may be revised after additional literature is considered. A data-processing procedure may be corrected when an inconsistency is discovered.
These changes are not necessarily problematic. Scientific inquiry frequently requires reconsideration as evidence becomes available.
The difficulty arises when consequential changes cannot be explained or reconstructed.
Documentation helps establish what changed, why the change was made, and whether it was supported by appropriate evidence.
When AI contributes to the process, a decision record can also clarify whether the system merely suggested a possibility, identified a problem that was independently verified, or generated material subsequently incorporated into the research.
The National Academies of Sciences, Engineering, and Medicine's report Reproducibility and Replicability in Science emphasizes the importance of transparent research practices and sufficient information to evaluate computational results.
Although the report does not prescribe a universal AI decision-log format, its principles support preserving records necessary to understand consequential analytical decisions.
Which AI-Influenced Decisions Are Important Enough to Record?
The need for documentation should depend on the decision's scientific significance, consequences, and applicable requirements.
A minor wording suggestion that does not change scientific meaning generally requires less documentation than a decision to replace the primary statistical model.
Researchers should consider whether the change affects the research question, protocol, sample, measurement, analysis, interpretation, or reported conclusions.
| AI-influenced activity |
Potential significance |
Suggested documentation |
| Minor grammatical revision |
Usually limited when meaning is unchanged |
Follow applicable disclosure or editing policies; a separate decision record may be unnecessary. |
| Change to a theoretical framework |
May alter the study's conceptual foundation |
Record the original framework, reason for reconsideration, supporting literature, and final rationale. |
| Modification of inclusion criteria |
May affect the study population and validity |
Record the proposed change, justification, approval status, and implementation details. |
| Correction to data transformation |
May affect analytical outputs |
Preserve relevant code versions, validation results, and affected outputs. |
| Change to primary statistical analysis |
May materially alter estimates and conclusions |
Record the original plan, identified limitation, verification, revised method, and authorization. |
| Revision of a causal interpretation |
May change the scientific claims made |
Record the original claim, evidence reviewed, alternative explanations, and revised conclusion. |
| Change to participant procedures |
May affect ethical or institutional obligations |
Record the rationale and obtain required approvals before implementation. |
These are practical recommendations rather than universal legal or publication requirements.
A journal may require disclosure of AI-assisted writing even when the edits are relatively minor. Conversely, an internal decision log may be advisable for a methodological change even when a journal does not explicitly request the log itself.
Distinguish AI Interaction Logs From Research Decision Logs
An AI interaction log records what occurred during communication with an AI system. A research decision log records why a consequential research decision was made.
These records overlap, but they are not interchangeable.
AI interaction log
Documents relevant AI tools, model information, prompts, outputs, dates, and other interaction details needed to understand how AI contributed.
Research decision log
Documents the original decision, issue identified, evidence evaluated, alternatives considered, final choice, responsible decision-makers, and justification.
A complete transcript may show that AI recommended a different analysis, but it may not reveal why the researcher accepted or rejected that recommendation.
Similarly, a decision record may explain the methodological change while omitting the AI contribution that initiated the reconsideration.
For consequential AI-influenced decisions, linking the relevant interaction to the decision rationale may provide a more useful audit trail than preserving an unstructured collection of conversations.
What Information Should a Research Decision Record Contain?
A practical record should be sufficiently detailed to explain and, where feasible, reconstruct the decision without becoming an administrative burden.
For a major AI-influenced decision, useful information includes:
-
Decision identifier and date: A unique reference and the date the decision was considered or approved.
-
Research stage: Whether the issue arose during design, data collection, analysis, interpretation, or manuscript preparation.
-
Original decision: The method, assumption, interpretation, or procedure initially planned or implemented.
-
AI contribution: What the AI system suggested, questioned, generated, or identified.
-
Tool information: The AI service, model or version identifier when available, and relevant settings or tool configuration.
-
Independent verification: The original evidence, methodological sources, calculations, or specialist opinions examined.
-
Final decision and rationale: What was accepted, rejected, or modified and why.
-
Consequences and authorization: Affected research outputs, responsible decision-makers, and any required approvals.
The record should also identify unresolved assumptions or limitations when they materially affect the decision.
Not every field will be relevant to every activity. The objective is to preserve the information necessary for meaningful review.
Why Is the Original Decision Important?
Without a record of the original decision, it may be difficult to establish what actually changed.
For example, a researcher may report that AI helped improve the statistical analysis. This statement does not explain whether AI introduced a new model, identified an incorrect assumption, or suggested a minor adjustment to an existing procedure.
Recording the original decision makes the change identifiable.
It also helps distinguish a planned methodological refinement from a post hoc analytical change made after examining results.
This distinction is particularly important when a study has been preregistered or includes a prespecified analysis plan.
Changes after observing the data may be scientifically justified, but they should not be presented as though they were specified before the results were known.
Document the Evidence, Not Just the AI Recommendation
A decision record should not consist solely of a copied AI response.
Generative AI can produce plausible explanations that contain incorrect assumptions, unsupported claims, or fabricated references.
If AI suggests that a statistical model is inappropriate, the record should identify how the researcher evaluated that concern.
For example, the researcher might inspect the data structure, consult methodological literature, reproduce calculations, or seek specialist review.
The distinction between an AI suggestion and independently established evidence is essential.
Researchers should document the evidence supporting consequential recommendations rather than treating AI-generated explanations as sufficient justification.
This allows later reviewers to understand why the decision was defensible even if the AI interaction is unavailable.
Preserve Relevant Prompts and Outputs When They Matter
Prompts and outputs may be important when they materially influence the research process.
For example, if AI generated analytical code that was incorporated into the study, preserving the relevant prompt, generated code, subsequent modifications, and validation results may help reconstruct the workflow.
If AI suggested a theoretical explanation that led to a major change in interpretation, the relevant exchange may clarify how the idea entered the research process.
However, preserving every conversational turn is not necessarily useful or appropriate.
Long transcripts can contain repeated questions, irrelevant responses, confidential information, and material unrelated to consequential decisions.
A curated record linked to relevant source materials may be more useful than an indiscriminate transcript archive.
Researchers should follow institutional retention rules and avoid preserving sensitive information unnecessarily.
Record the AI Tool and Model as Precisely as Possible
AI services may change their models, interfaces, and available capabilities over time.
A record stating only that ChatGPT or another general-purpose AI service was used may be insufficient to reconstruct a technically consequential interaction.
Where available, record the model identifier, relevant version information, date of use, important configuration settings, and whether external tools or retrieval features were involved.
Some consumer AI interfaces do not expose every underlying model detail. Researchers should document what can be reliably established rather than inventing unavailable information.
Model identification alone does not guarantee reproducibility. Outputs may vary because of nondeterministic generation, changing infrastructure, external information sources, or differences in the interaction context.
For consequential workflows, preserve the actual outputs used and independently verified downstream artifacts.
The distinction between recording the model and preserving enough information to reconstruct its contribution becomes especially important in long-running projects.
How Should Researchers Document AI-Generated Code Changes?
AI-generated code can materially affect research results even when the researcher does not adopt an AI-generated interpretation.
Suppose AI suggests changing a data-cleaning function or modifying a regression specification. The resulting code may alter the dataset, estimates, or figures.
A defensible record should preserve the relevant code versions, identify the changed operations, and document tests establishing that the revised implementation behaves as intended.
Version-control systems can help track code modifications, although version history alone does not explain the scientific justification for a change.
Researchers should link technical changes to the relevant analytical decision and verification evidence.
For example, a commit message stating "updated analysis" provides less information than a record explaining that the revision corrected an incorrectly coded outcome variable and that the corrected results were independently reproduced.
Why Are Preregistration and Analysis Plans Particularly Important?
Preregistration can distinguish decisions made before observing outcomes from those developed after results become available.
The Center for Open Science's preregistration resources explain how researchers can document research questions, hypotheses, and analysis plans before conducting the relevant analyses.
Preregistration does not prohibit justified deviations. However, deviations should be transparently identified and explained.
If AI prompts a change to a prespecified analysis, the researcher should document when the change occurred, what information was available, and why the revision was considered necessary.
Where applicable, the original plan should remain accessible alongside the revised procedure.
A post hoc change should not be retrospectively represented as a prespecified decision.
This principle applies regardless of whether the change originated from AI, a supervisor, a statistician, or the researcher's own reconsideration.
Internal Documentation Is Not the Same as Formal AI Disclosure
Researchers should distinguish maintaining internal records from disclosing AI assistance in a manuscript or research report.
An internal log may contain detailed prompts, analytical decisions, verification steps, and discussions among collaborators.
A formal disclosure may instead summarize which AI tool was used and how it contributed, according to the requirements of the journal, institution, funder, or other responsible organization.
The International Committee of Medical Journal Editors provides recommendations concerning disclosure of AI assistance and the responsibilities of human authors. Other publishers and institutions may impose different requirements.
Maintaining a detailed internal log does not automatically satisfy a required publication disclosure.
Likewise, including an AI-use statement in a manuscript does not necessarily provide sufficient information to reconstruct a consequential analytical change.
Researchers should check the applicable policies and treat documentation and disclosure as related but distinct responsibilities.
Who Should Maintain the Decision Record in Collaborative Research?
In collaborative projects, responsibility for documenting decisions should be assigned clearly.
The principal investigator, project coordinator, designated analyst, or another responsible team member may maintain the central decision log, depending on the project's governance arrangements.
Individual contributors should also preserve relevant technical records for the work they perform.
For example, a statistician may retain analysis scripts and validation results, while the research team maintains the record explaining why the analytical approach changed.
A shared decision record can help prevent inconsistent versions of the study rationale from circulating among collaborators.
Access should be controlled according to the sensitivity of the information and the team's legitimate responsibilities.
Protect Confidentiality and Research Data During Documentation
Documentation can itself create privacy and security risks.
AI interaction transcripts may contain participant information, unpublished findings, confidential manuscripts, or restricted research materials.
Researchers should not assume that preserving every prompt is appropriate simply because it could improve transparency.
Data minimization, access controls, approved storage systems, and applicable retention requirements should guide what is preserved.
Where possible, decision records can refer to securely stored source materials rather than duplicating sensitive content.
De-identification should be evaluated carefully because combinations of contextual details may still permit identification.
Watch Out
Do not upload confidential research materials to an unapproved AI service or store sensitive prompts in an unrestricted decision log. Documentation must comply with applicable ethics approvals, privacy requirements, contractual obligations, and institutional information-security policies.
Can a Decision Log Prove That the Research Was Correct?
No. Documentation improves traceability, but it does not independently establish scientific validity.
A researcher may thoroughly document an incorrect decision. Conversely, a scientifically defensible decision may be poorly documented.
The value of a decision log is that it makes the reasoning and evidence available for examination.
Independent verification remains necessary for consequential claims.
Researchers should therefore treat documentation as one component of responsible research practice, alongside methodological rigor, appropriate review, and accountable decision-making.
When Should a Researcher Update the Decision Log?
Consequential decisions should ideally be documented close to the time they occur.
Waiting until manuscript submission may make it difficult to reconstruct the original reasoning, identify which AI outputs were used, or distinguish planned from post hoc changes.
A practical approach is to update the log whenever a material decision is proposed, approved, implemented, or subsequently revised.
Records can also identify pending decisions and unresolved verification requirements.
This creates a chronological account of how the research developed rather than a retrospective narrative assembled after the final results are known.
