03 · What You Need to Know
Which AI Settings Matter for Research Reproducibility?
What Are AI Tool Settings?
AI tool settings are configurable parameters or options that influence how an artificial intelligence system processes inputs or generates outputs.
Depending on the system, they may control sampling behavior, output length, response format, randomization, retrieval, or other aspects of the workflow.
For example, a researcher using an AI application programming interface may specify a temperature value or maximum output length.
Another researcher using a consumer chatbot may have no access to those controls.
Settings should therefore be understood in relation to the actual system used rather than treated as a universal collection of parameters available in every AI tool.
Why Can AI Settings Affect Research Outputs?
Some AI systems generate responses through probabilistic processes.
Settings can influence which outputs are selected, how much variation is permitted, or how long a response may be.
For example, changing sampling parameters may alter the wording of generated explanations or the categories assigned to ambiguous text.
Output limits may truncate responses, while structured-output controls may constrain the format of generated data.
These effects can matter when AI outputs become research observations or analytical inputs.
However, the influence of a particular setting depends on the model, task, and implementation.
Researchers should not assume that every parameter affects every AI system in the same way.
What Is Temperature in Generative AI?
Temperature is a parameter used in many generative language models to modify the probability distribution from which output tokens are sampled.
Lower temperature values generally make higher-probability tokens more dominant, while higher values typically allow greater variation in token selection.
However, the exact interpretation and permitted range depend on the model and provider.
Temperature should not be described simply as a universal creativity setting.
For example, a higher temperature may produce more varied wording, but it does not necessarily produce more original or scientifically useful ideas.
Likewise, lower temperature does not establish factual accuracy.
Does Setting Temperature to Zero Guarantee Identical Responses?
No. A temperature value of zero may reduce sampling variation in systems that support that setting, but it does not universally guarantee deterministic outputs.
Differences may still arise from model implementation, computational infrastructure, hidden context, service updates, or other factors.
Some providers may also define or implement temperature differently.
Researchers should therefore avoid describing temperature zero as proof that an AI-assisted procedure is fully reproducible.
Where identical outputs are important, the researcher should evaluate actual repeatability under the relevant system configuration.
Preserving the original outputs remains valuable even when low-variability settings are used.
What Is Top-p Sampling?
Top-p, also called nucleus sampling, is a method that restricts token selection to a set of candidates whose cumulative probability reaches a specified threshold.
It can influence output diversity by changing the range of tokens considered during generation.
However, the exact behavior depends on the implementation.
Some providers recommend adjusting either temperature or top-p rather than modifying both simultaneously.
Researchers should follow the relevant system documentation and record the values actually used when they matter to the methodology.
Top-p should not be interpreted as a direct measure of response accuracy or scholarly reliability.
What Is an Output Token Limit?
An output token limit constrains the maximum amount of generated content a model may produce in a response.
Depending on the system, the parameter may be described as maximum output tokens or another provider-specific name.
The limit can affect whether an AI response is complete.
For example, a model asked to classify hundreds of survey responses may stop before processing every item if the output limit is insufficient.
Researchers should therefore consider whether truncation affected the procedure.
Recording the output limit may be important when it influences the completeness or structure of generated research data.
What Is a Random Seed?
A random seed is a value used to initialize certain pseudorandom processes.
Some AI systems or software environments expose seed controls intended to support more consistent generation or experimental repeatability.
However, the availability and effectiveness of seed controls vary across systems.
Using the same seed does not necessarily guarantee identical outputs across different hardware, model versions, or provider configurations.
Researchers should document the seed where it is available and methodologically relevant.
They should also evaluate whether the particular system provides deterministic or only best-effort repeatability.
Which Settings Are Most Relevant to Record?
| Setting or Configuration |
Potential Effect |
Documentation Consideration |
| Temperature |
May influence variation in token selection. |
Record the value where available and relevant. |
| Top-p |
May affect the candidate token distribution. |
Document the configured value if used. |
| Output token limit |
May constrain response length or completeness. |
Record when truncation or output structure matters. |
| Random seed |
May influence repeatability in supported systems. |
Record the seed and relevant system limitations. |
| Structured-output configuration |
May constrain response format or permitted values. |
Preserve schemas or configuration rules when consequential. |
| Retrieval configuration |
May influence which external information enters the response. |
Document relevant sources, indexes, or retrieval settings. |
| System instructions |
May influence model behavior and task interpretation. |
Preserve instructions under researcher control where relevant. |
| Tool or model version |
May affect available capabilities and output behavior. |
Record verified identifiers and relevant dates. |
Not every system exposes all these settings, and not every research task requires them.
The appropriate documentation should reflect the actual procedure.
Are AI Settings the Same as Model Hyperparameters?
Not necessarily. The terms may overlap in informal discussion but refer to different levels of configuration.
Model hyperparameters can describe choices made during model training, such as learning rates, architecture settings, or optimization procedures.
Inference settings govern how a trained model is used to generate outputs.
For example, temperature and output limits are commonly associated with inference rather than model training.
Researchers should distinguish training configurations from inference controls when reporting computational methods.
This distinction becomes particularly important in studies that develop or fine-tune models rather than merely use existing services.
Do Consumer Chatbots Expose Their Settings?
Not always. Consumer chatbot interfaces may hide sampling parameters, system instructions, routing decisions, or other technical details.
Researchers may be able to select a model or interaction mode without controlling the underlying generation settings.
These limitations should be recognized when the application is used for consequential research procedures.
Researchers should not assume that a hidden parameter has a particular default value unless that information is documented for the relevant service and time.
Where exact settings are unavailable, the research record should state that limitation.
Preserving prompts, outputs, dates, and available model information may still support methodological transparency.
What If the Researcher Uses an API?
Application programming interfaces may provide more explicit configuration controls than consumer chat interfaces.
Depending on the provider and model, researchers may be able to specify sampling parameters, output formats, tool access, and other options.
These settings can be recorded through configuration files, scripts, or structured request metadata.
However, not every API supports identical parameters or guarantees deterministic behavior.
Researchers should consult the relevant documentation and preserve the actual configuration used.
Where the AI-assisted procedure is central to the study, retaining executable scripts or request specifications may support later evaluation.
Should Researchers Record Default Settings?
Where default settings are documented and relevant, recording that the system used them may be useful.
However, researchers should distinguish a known default from an assumed one.
For example, a provider may publish a default value for a particular API parameter, while a consumer application may not disclose its underlying configuration.
The researcher should not assign the API default to a chatbot interface without evidence that the same configuration applied.
Where the default is unknown, the record should state that the setting was not exposed or not recorded.
Accurate uncertainty is preferable to unsupported technical precision.
Can Settings Change During a Research Project?
Yes. Researchers may modify parameters during experimentation or switch between different tools and configurations.
For example, a study may begin with one temperature setting and later adopt another after observing inconsistent classifications.
Such changes may affect the comparability of outputs.
Where settings materially influence the analysis, researchers should record which configuration was used for each relevant batch or stage.
The final manuscript should not imply that one stable configuration governed the entire procedure if consequential changes occurred.
Documenting revisions can help explain differences in generated outputs.
Should Settings Be Recorded for AI-Assisted Qualitative Coding?
Where settings are available and influence the coding procedure, recording them may support methodological evaluation.
For example, a researcher using an API to classify interview excerpts may specify sampling parameters and a structured output schema.
These settings can affect how classifications are generated or formatted.
However, recording settings does not establish that the coding framework is valid.
Researchers must still evaluate the classifications against the data and chosen qualitative methodology.
Technical repeatability and interpretive credibility are related but distinct concerns.
Should Settings Be Recorded for AI-Generated Statistical Code?
Settings may help document the code-generation interaction, but the final tested code is generally more directly relevant to reproducing the analysis.
For example, a researcher may use an AI model to generate preliminary R code and subsequently revise it before execution.
The final script, software dependencies, data transformations, and execution environment determine the reported computations.
Recording AI generation settings may provide useful provenance, but it does not replace documentation of the analytical implementation.
Researchers should distinguish reproducing the AI response from reproducing the statistical analysis.
What About Retrieval-Augmented Generation Settings?
Retrieval-augmented generation systems combine model-generated responses with information retrieved from external sources.
Outputs may depend on the documents indexed, retrieval method, ranking procedure, number of retrieved passages, and other configuration choices.
For example, an AI literature assistant may produce different summaries depending on which publications were retrieved.
When such systems form part of a research method, documenting the retrieval configuration may be more important than recording sampling temperature alone.
Researchers should identify which external information sources and retrieval procedures influenced the outputs.
Where indexes or databases change over time, the date and available source records may also be relevant.
What About AI Image Generation Settings?
Image-generation systems may expose settings such as dimensions, aspect ratio, random seed, guidance controls, or other model-specific parameters.
These settings can influence the generated visual material.
Where AI-generated images are used as experimental stimuli or form part of the research methodology, relevant settings should be documented.
However, not every image-generation service exposes the same controls.
Researchers should preserve the actual images used in the study because repeating the generation procedure may not reproduce them exactly.
Publisher restrictions on AI-generated imagery must also be considered separately from technical documentation.
Does Recording Settings Guarantee Reproducibility?
No. Even a complete record of visible settings may not capture every factor influencing an AI system's output.
Commercial services may use hidden instructions, provider-side processing, or changing infrastructure.
Some operations may also be nondeterministic.
Researchers should therefore distinguish procedural documentation from guaranteed output replication.
Where exact reproduction is not possible, preserving the actual outputs and evaluating repeatability may be more informative than claiming determinism.
Model and setting records should be interpreted as components of a broader reproducibility strategy.
How Should Settings Be Documented in an AI-Use Log?
A structured AI-use log can connect configuration information to the relevant task and outputs.
For example, a log entry may identify the model, prompt template, configured sampling parameters, input dataset, output file, and verification procedure.
Where settings are unavailable, the log should indicate that they were not exposed.
The record should not include invented values merely to complete every field.
For complex workflows, configuration files or scripts may provide a more reliable record than manually transcribed values.
The goal is to preserve the actual execution context as accurately as possible.
Should Settings Appear in the Published Manuscript?
That depends on the journal and the methodological significance of the AI procedure.
For a study evaluating AI model performance, configuration details may be central to understanding the findings.
For routine manuscript-language assistance, technical settings may not be required.
Where settings materially affect analysis, they may belong in methods, supplementary materials, or an associated repository.
Researchers should distinguish internal documentation from publication disclosure.
The question of what an AI disclosure statement should include should be resolved according to the journal's requirements and the role of AI in the study.
How Long Should AI Settings Records Be Retained?
Retention depends on the applicable research-records policy and the importance of the settings to the published findings.
Where configuration information is necessary to reconstruct a consequential AI-assisted procedure, it may need to be retained with the broader research archive.
However, no universal retention period applies specifically to every AI setting.
Researchers should preserve relevant configuration records for the period required by their institution, funder, or research agreement.
Records should remain connected to the outputs and procedures they describe.
What If Settings Were Not Recorded During the Research?
Researchers may consult saved scripts, configuration files, API logs, or other contemporaneous records.
However, they should not infer hidden settings from the current interface or assume that present defaults applied historically.
If relevant information cannot be recovered, the limitation should be documented.
For example, a researcher may report that AI-assisted classifications were generated through a consumer application whose sampling parameters were not exposed.
That statement is more accurate than assigning an unsupported temperature value.
Where the missing configuration materially limits reproducibility, the limitation should be considered when interpreting the findings.
