03 · What You Need to Know
How to Identify and Document AI Systems Used in Scientific Research
Why the Application Name May Be Insufficient
An AI application is not necessarily identical to the underlying model that generates its responses.
A product may provide access to several models, route requests between systems, or incorporate additional services such as retrieval and external tools.
The underlying components may also change during the research project.
Consequently, reporting only the application name may leave important aspects of the analytical procedure unidentified.
For incidental editorial assistance, the application name and a description of its role may satisfy the applicable reporting requirements.
For AI-generated research measurements or classifications, more detailed identification may be necessary to support scientific evaluation.
Distinguish the Provider, Product, Model, and Deployment
Researchers should distinguish several elements that are sometimes treated as interchangeable.
AI application or product
The interface or service through which a researcher accesses AI capabilities.
AI model
The underlying computational model used to generate predictions, classifications, text, or other outputs.
The provider is the organization offering the relevant service or model. Deployment describes how the system is made available, such as through a hosted application, API, or locally controlled environment.
These distinctions matter because two researchers may use the same application but different model versions or configurations.
Likewise, the same model may behave differently when embedded in systems with different instructions, retrieval mechanisms, or connected tools.
Which Model Details Should Researchers Record?
| Documentation element |
Scientific purpose |
Example of information to preserve |
| Provider |
Identifies the organization responsible for the service |
Provider name |
| Application or interface |
Identifies how the model was accessed |
Web application, API, or local software |
| Model identifier |
Distinguishes the model selected or reported |
Available model name or identifier |
| Version or snapshot |
Supports identification of a particular release |
Published snapshot identifier, if available |
| Execution date |
Places processing within the system's development history |
Date of each consequential analytical batch |
| Generation settings |
Documents relevant output controls |
Available sampling parameters or settings |
| Instructions |
Explains how the task was specified |
Prompt, coding criteria, system instructions where accessible |
| Input-data version |
Identifies the material processed |
Secure dataset reference or version identifier |
| Connected components |
Identifies external dependencies |
Retrieval service, connected database, or tool |
| Original outputs |
Preserves what the system actually generated |
Classification file, code, or extracted evidence |
The appropriate documentation depends on the task. Researchers should not assume that every field is available or necessary for every AI interaction.
What Is a Model Identifier?
A model identifier is a name or reference used to distinguish a particular AI model or model offering.
Some providers expose identifiers through their APIs or technical documentation.
Others display general model names through a user interface without identifying the exact underlying release.
Researchers should preserve the identifier actually available at the time of use.
A general product label should not be represented as an exact model snapshot unless the provider establishes that relationship.
Why Model Versions and Snapshots Matter
AI providers may update models to change capabilities, performance, safety behavior, or other characteristics.
These changes can affect research outputs even when prompts remain unchanged.
For example, two model versions may classify the same ambiguous student reflection differently.
If those classifications become research variables, the update may affect measurement consistency.
Recording available version information helps researchers identify whether different outputs were generated under different conditions.
However, version documentation alone does not establish that results remain scientifically comparable.
What if the Provider Does Not Disclose the Exact Version?
Some hosted AI services do not provide complete model identifiers or version histories to users.
Researchers should not invent missing technical information or infer a precise model version from the date alone.
Instead, they should preserve the application name, displayed model information, execution date, relevant configuration, and original outputs.
Where the limitation affects methodological reproducibility, it should be acknowledged.
For consequential tasks requiring exact model identification, researchers should consider whether a different deployment or validated procedure would be more suitable.
Why Execution Dates Should Be Preserved
Execution dates can help identify when a model-dependent procedure occurred.
For example, a longitudinal project may classify responses at the end of each semester.
If the provider updates its system between semesters, the processing dates can help investigators identify which batches may have been affected.
Dates do not establish the exact underlying model when version information is unavailable.
They nevertheless provide useful context for investigating changes and interpreting archived results.
Which Configuration Settings Matter?
Some AI systems expose parameters controlling aspects of generation, including sampling behavior or output limits.
Other settings may be managed internally by the provider.
Researchers should record available settings that materially influence consequential outputs.
For example, a model used for structured classification may be configured differently from one used for exploratory writing.
However, identical settings do not necessarily guarantee identical responses.
Researchers should document relevant controls without claiming that they eliminate every source of output variability.
Why Prompts Are Part of Model Documentation
Prompts and instructions may define how an AI model performs a research task.
For example, a classification prompt may specify category definitions, examples, exclusions, and how ambiguous responses should be handled.
Changing these instructions can alter the generated labels even when the model remains unchanged.
For consequential procedures, researchers should preserve the relevant prompts and any material revisions.
However, prompts should not be treated as a complete description of the system when additional instructions or connected components affect processing.
What About System Instructions and Hidden Configuration?
Some AI applications use system-level instructions or provider-managed configurations that are not visible to researchers.
These components may influence model behavior.
Researchers should distinguish accessible settings from inaccessible dependencies.
They should not claim to have documented the complete computational environment when important components remain unknown.
For model-dependent research, such limitations may affect the strength of reproducibility claims.
Preserving original outputs and independent validation evidence can help support later evaluation.
How Should Researchers Document API-Based AI Use?
API-based workflows may provide more direct access to model identifiers, request parameters, and structured outputs than some graphical interfaces.
Researchers should preserve relevant request configurations, model identifiers, execution records, and generated artifacts.
They should also document software dependencies and any preprocessing or postprocessing performed outside the model.
However, API access does not automatically guarantee complete version control or deterministic behavior.
Researchers should evaluate the actual service guarantees and available documentation.
How Should Locally Deployed Models Be Documented?
Locally deployed models may allow researchers to preserve model files, configuration details, and software environments.
Relevant documentation may include model identifiers, source repositories, revision information, runtime dependencies, and available file checksums.
Researchers should also record relevant preprocessing, inference, and postprocessing procedures.
A preserved model file may improve reconstructability, but exact reproduction can still depend on hardware, software libraries, and computational conditions.
Local deployment should not be treated as automatically reproducible without appropriate documentation and testing.
What if the AI System Uses Retrieval or External Tools?
Some AI systems generate responses using retrieved documents, connected databases, search services, or computational tools.
These components may materially influence the output.
For example, an AI-generated literature summary may depend on the documents retrieved at a particular time.
Researchers should identify relevant retrieval sources, document versions, and tool operations where feasible.
Recording only the underlying language model may be insufficient to reconstruct the procedure.
The documentation should reflect the complete workflow rather than one component in isolation.
How Should Researchers Handle Model Changes During Long Projects?
Longitudinal or multi-stage research may involve different model versions across data collection or analysis periods.
Researchers should record which available model identifiers were associated with each consequential processing batch.
Where a model update may affect research measurements, additional validation may be necessary.
For example, a classifier used across several semesters should not automatically be assumed to apply category definitions consistently after an update.
The research team should assess comparability and document any reprocessing or methodological adjustments.
Does Documenting the Model Guarantee Reproducibility?
No. Model documentation and computational reproducibility are related but distinct.
A preserved model identifier may help identify the system used, but the original model may later become unavailable.
Even when access remains possible, stochastic generation and other dependencies may produce different outputs.
Researchers should preserve original consequential outputs and relevant analytical artifacts.
For downstream calculations, independently executable code and data may provide stronger reproducibility evidence than attempting to regenerate an earlier AI conversation.
How Much Model Documentation Is Necessary?
The required detail should reflect the scientific role of the AI system.
Incidental language assistance may require a concise description consistent with publication policy.
AI-generated research variables, classifications, or primary analytical outputs generally warrant more detailed technical and methodological records.
Researchers should ask whether another qualified person could identify the relevant procedure and evaluate its contribution to the findings.
Documentation should be sufficient for the intended scientific purpose without creating unnecessary administrative or confidentiality burdens.
What if Exact Identification Is Impossible?
Researchers should distinguish incomplete documentation from an automatically invalid research result.
For some supporting tasks, limited model information may have little effect on the defensibility of the study.
For primary measurement procedures, inability to identify the system may create substantial uncertainty.
The team should assess whether preserved outputs, validation evidence, and other records adequately support the findings.
Where essential reconstructability cannot be established, the procedure may need to be replaced or its limitations explicitly reported.
