Can AI Coding Change When the Same Prompt Is Rerun?

Why Does AI Sometimes Assign Different Codes to the Same Response?

You ask generative AI to classify a student's survey response using a predefined research codebook. The model assigns "institutional support." You run the same prompt again, with the same response and coding instructions, but this time it assigns "technical assistance."

Nothing appears to have changed. The question, response, and codebook are identical. Why did the classification change?

This matters because researchers may assume that a standardized prompt produces standardized coding decisions. If repeated executions yield different assignments, the resulting dataset may depend partly on when and how the model was run rather than solely on the research evidence.

Yes, Identical Prompts Can Produce Different Coding Decisions

Why AI Coding Changes Across Repeated Executions

What Does It Mean for AI Coding to Be Nondeterministic?

A deterministic coding procedure produces the same output whenever the same input is processed under identical conditions. A nondeterministic procedure can produce different outputs across executions.

Many generative language models select outputs using probabilistic generation mechanisms. Depending on the system and its configuration, different tokens may be selected from a distribution of possible continuations.

For research coding, this means that a model may assign one category during an initial execution and another category during a later execution, particularly when the response supports several plausible interpretations.

However, not every difference is caused by random sampling. Model updates, changes in hidden instructions, software infrastructure, and input context may also affect the result. Researchers should distinguish output variability under controlled conditions from changes in the system itself.

What Is the Difference Between Prompt Stability and Coding Stability?

A prompt can remain unchanged while its outputs vary. Conversely, two differently worded prompts may sometimes produce identical classifications.

Prompt stability is a condition researchers can attempt to control. Coding stability is an empirical property that must be observed.

A model may follow the same instructions yet make different decisions about an ambiguous response. Researchers should therefore evaluate the resulting assignments rather than assuming that identical wording establishes reproducibility.

Does Temperature Affect AI Coding Consistency?

Temperature is a generation parameter supported by some language-model systems. It influences how probability differences among possible output tokens affect token selection.

In systems that expose conventional temperature controls, lower settings generally reduce sampling variability, while higher settings tend to permit more diverse outputs. The exact behavior depends on the model and implementation.

For structured research classification, lower-variability settings may be useful because researchers usually want the model to apply established definitions rather than generate creative alternatives.

However, temperature zero does not universally guarantee identical outputs. Numerical computation, model-serving behavior, software changes, and other implementation details can still affect results. Some newer models also restrict or do not expose temperature settings.

Researchers should consult the official documentation for the specific model and service rather than assuming that all generative AI systems support the same controls.

Can Model Updates Change Coding Results?

Yes. A model provider may update a deployed system, replace a model version, modify supporting instructions, or change processing infrastructure. These changes can influence classification behavior even when researchers reuse an earlier prompt.

For example, a model might initially interpret "lack of support" as inadequate administrative assistance. A later version could favor technical support because its classification behavior has changed.

Researchers should record the model identifier and available version information. When a provider offers versioned model snapshots, using a fixed snapshot can reduce one source of variation, although it does not guarantee complete determinism.

When only a general product name is available, researchers should document the service, execution date, and relevant configuration as accurately as possible.

Can Conversation History Change AI Coding?

Yes. A prompt submitted within an ongoing conversation may be interpreted in light of earlier exchanges, examples, corrections, or instructions.

Suppose a researcher initially asks AI to classify several statements about institutional support. Later in the conversation, the researcher corrects the model and emphasizes technical assistance. A subsequent classification may be influenced by those earlier corrections.

The researcher might believe that the same coding prompt has been rerun, but the full input context is no longer identical.

For controlled classification, independent requests with explicitly supplied instructions may be easier to document than a long conversation containing changing context. This is particularly relevant when coding thousands of responses across multiple batches.

Can Response Order Affect AI Coding?

Potentially. When several responses are submitted together, the surrounding examples may influence how the model interprets a particular statement.

Consider the response "I could not keep up with the lessons." When placed among comments about unstable internet, the model may associate it with connectivity. When placed among comments about excessive assignments, it may favor workload difficulties.

The statement itself does not identify either cause. The appropriate classification may therefore be unresolved unless additional context is available.

Researchers can test order sensitivity by presenting the same benchmark responses in different batch positions while preserving their original content and coding rules.

Such tests help determine whether classifications depend on neighboring material rather than only on the intended coding evidence.

Are Ambiguous Responses More Likely to Produce Different Codes?

They may be, particularly when several categories appear plausible and the coding framework does not establish a clear decision rule.

For example, "Nobody helped us understand the new system" might receive "insufficient training," "institutional support," or "technical assistance," depending on how those categories are defined.

However, repeated disagreement is not a complete measure of ambiguity. A model may consistently assign an unsupported category, while a clear response may occasionally receive different outputs because of processing variability.

Researchers should distinguish the stability of the model's behavior from ambiguity in the research material.

How Can Researchers Measure Repeat-Run Coding Stability?

For structured classification, researchers can repeat the same coding procedure and compare the resulting assignments.

One straightforward measure is exact-match agreement, which counts the proportion of responses receiving identical code assignments across two executions.

Researchers can also examine category-specific changes, confusion matrices between runs, and agreement statistics suited to the coding design.

When more than two executions are available, researchers should specify how stability is summarized. Pairwise agreement, agreement across all runs, and the proportion of responses that ever change codes answer different questions.

How Many Times Should Researchers Rerun the Same Coding Task?

There is no universally required number of repetitions. The appropriate design depends on the study's purpose, expected variability, computational resources, and consequences of unstable assignments.

Two executions provide an initial comparison but may fail to reveal less frequent variation. Additional runs can provide more information about how often classifications change under the tested conditions.

Researchers should select repetitions prospectively where possible and avoid repeatedly running a prompt until a preferred answer appears.

Repeated testing should also use a sufficiently varied set of responses. Testing only straightforward examples may produce high stability while overlooking instability in the cases that matter most.

Should Researchers Use Majority Voting Across Repeated Runs?

Majority voting assigns the code selected most frequently across repeated executions. For example, if a response receives code A in four of five runs and code B in one, the procedure selects code A.

This can reduce some forms of output variability, but it does not necessarily improve validity. Repeated executions may share the same underlying bias or misunderstanding.

Majority voting also introduces additional methodological choices, including the number of runs, tie-breaking rules, and handling of multi-label classifications.

Researchers should evaluate whether aggregation improves performance against independently reviewed reference decisions rather than assuming that the most frequent AI answer is correct.

Does Stable AI Coding Mean the Coding Is Valid?

No. A model may repeatedly assign the same incorrect category because its interpretation is systematically flawed.

Suppose AI consistently interprets statements about institutional scheduling constraints as poor student time management. Even perfect agreement across repeated executions would not correct that conceptual error.

The distinction between coding consistency and coding validity remains essential. Stability describes reproducibility of assignments under specified conditions, whereas validity concerns whether the assignments adequately represent the intended concepts.

Researchers should therefore evaluate repeat-run stability alongside coding accuracy, category-specific errors, and methodological appropriateness.

What Does Research Suggest About LLM Output Variability?

Research on language-model reproducibility has demonstrated that repeated executions and apparently controlled generation settings do not always guarantee identical outputs.

Ouyang and colleagues (2023), in "Non-Determinism of 'Deterministic' LLM Settings," investigated sources of output variation under deterministic generation configurations. Their work illustrates why controlling sampling parameters does not necessarily eliminate implementation-level variability.

Gilardi, Alizadeh, and Kubli (2023) evaluated ChatGPT on text-annotation tasks and examined agreement across repeated classifications under different temperature settings. Their work demonstrates the importance of measuring annotation stability empirically.

These findings support documenting model configurations and testing the actual coding procedure. They do not establish a universal instability rate for qualitative research coding.

How Should Researchers Document Prompt Reruns?

Researchers should preserve the exact prompt, codebook version, input records, model identifier, available generation settings, execution date, and resulting assignments.

If the model supports a seed or another reproducibility parameter, that information should also be recorded. However, a fixed seed should not be treated as a guarantee of identical outputs unless the provider explicitly supports that guarantee under the relevant conditions.

When the coding procedure changes, researchers should document the reason and distinguish results produced under different configurations.

These records contribute to reproducible AI-assisted coding, even when exact output replication remains technically uncertain.

Testing Whether AI Assigns the Same Codes Across Five Runs

Step 1: Establish Controlled Conditions

The researcher preserves the same input records, coding definitions, prompt instructions, model version where available, and generation settings.

Five separate executions are conducted, with each response receiving one code during each run.

Step 2: Compare the Results

Suppose the hypothetical results show the following:

Eighty percent of responses receive the same code in every run. The remaining 20% change at least once.

This measure is stricter than comparing only two runs because it requires agreement across all five executions.

Step 3: Examine the Unstable Responses

The researcher inspects the 40 responses that changed classifications.

Suppose many involve overlapping categories or insufficient context. The researcher examines whether the codebook needs clearer boundaries or whether the responses should be flagged for review.

However, the researcher also independently reviews a selection of stable responses. Some may be consistently incorrect.

Step 4: Evaluate the Implications

If unstable assignments affect categories central to the study, the researcher may revise the coding procedure or increase human review.

Any revised procedure should be evaluated on additional material rather than accepted solely because it produces more stable outputs on the examples used during refinement.

The result shows that repeatability is an observable property of the coding workflow, not something guaranteed by repeating the same prompt.

Misconceptions About Repeated AI Coding

How to Make Repeat-Run Testing Useful for Your Research

Researchers should test output stability when coding decisions may be sensitive to model behavior or when the results must be reproduced across processing sessions.

Researchers should avoid treating stability as the only quality criterion. A model that consistently assigns incorrect labels may be operationally predictable but analytically unsuitable.

Where instability is substantial, the appropriate amount of manual verification may need to increase, particularly for categories affected by repeated disagreement.

Before Testing AI Coding Across Repeated Prompts

Frequently Asked Questions About AI Coding Across Prompt Reruns

The Same Prompt Does Not Guarantee the Same Research Code

How to Cite This Guide

This guide is intended to be read, shared, and used in research, teaching, and academic work. If you draw on its ideas, explanations, or other content, please acknowledge the source by citing the guide. Doing so gives appropriate credit and helps your readers locate the original resource.