03 · What You Need to Know
When and How Repeated AI Executions Improve Research Evaluation
Why One AI Execution May Be Insufficient
Many generative AI systems produce outputs through processes that permit variability. A model may assign different labels to the same text, extract different values from an ambiguous document, or emphasize different evidence when interpreting a research finding.
A single execution reveals what the system produced on that occasion. It does not establish how consistently the system would perform the task across repeated executions.
This matters when the output becomes part of the research evidence. If a classification enters a statistical analysis, an alternative classification could change the resulting estimates. If an extracted value enters a meta-analysis, an incorrect or inconsistent extraction could influence the pooled result.
Repeated execution is one way to investigate this uncertainty. It is not the only method of validation, and it should not replace checking outputs against the original evidence.
Which AI-Assisted Tasks Are Most Worth Rerunning?
The strongest justification arises when AI makes decisions that materially influence research results and when those decisions may vary across executions.
| AI-Assisted Task |
Why Repetition May Help |
What Else Is Needed |
| Qualitative coding |
Identifies inconsistent category assignments |
Documented coding framework and independent evaluation |
| Research article screening |
Reveals unstable inclusion or exclusion recommendations |
Verification against eligibility criteria and source records |
| Numerical data extraction |
Detects differences in extracted values |
Direct checking against the original documents |
| Text classification |
Measures consistency of generated labels |
Reference labels and category-specific error analysis |
| Interpretive analysis |
Reveals alternative interpretations or unsupported claims |
Independent examination of evidence and interpretive validity |
| Routine language editing |
Usually offers limited methodological value |
Verification of the final text and applicable disclosure |
Repeated runs may be particularly valuable when the model's outputs are treated as measurements rather than suggestions.
For low-risk writing assistance, repeating the same request may provide alternative wording but is unlikely to establish anything important about the validity of the underlying research.
What Should Remain Constant Across Repeated Runs?
To evaluate ordinary output variability, researchers should hold the available execution conditions constant.
These conditions may include the model identifier, prompt version, input data, generation settings, output schema, and relevant contextual instructions.
Researchers should also distinguish independent executions from repeated questions within one continuing conversation. Previous messages may affect later responses, making the complete inputs different even when the final user message is identical.
Where available, temperature and other generation settings should be documented because they may influence output variability.
If the model or prompt changes between runs, the exercise becomes a comparison of different analytical conditions rather than a straightforward test of repeatability.
Even under apparently equivalent conditions, the same prompt may produce different responses.
How Many Times Should Researchers Rerun an AI Task?
There is no universally defensible answer such as three, five, or ten runs.
The number of executions should reflect what the researcher intends to estimate. Two runs permit a basic comparison, but they provide limited information about the range or frequency of possible outputs.
Additional runs can reveal variation that was not observed in the first comparison. However, increasing the number of executions does not automatically correct systematic errors or compensate for an unrepresentative evaluation dataset.
Researchers should consider the expected variability, precision of the stability estimate, number of observations, category distribution, computational cost, and consequences of disagreement.
A small pilot may help estimate variability before designing a more substantial evaluation. Where uncertainty estimates are important, researchers should consider confidence intervals or other methods appropriate to the repeated-measures design.
Because repeated outputs from the same model and input are related, they should not automatically be treated as independent observations in statistical calculations.
Repeating the Same Input Is Different From Testing Many Inputs
Researchers need to distinguish the number of observations evaluated from the number of times each observation is processed.
Running one interview excerpt 100 times may reveal how the model behaves on that excerpt. It does not establish stability across an entire collection of interviews.
Conversely, processing 1,000 different excerpts once each provides broad coverage but little direct evidence about repeated-run consistency.
A useful evaluation design considers both dimensions. Researchers may select a representative sample of observations and process each observation multiple times.
Sampling should reflect the intended research population and include relevant variation, such as ambiguous responses, uncommon categories, or different linguistic characteristics.
For example, a study analyzing multilingual student feedback should not infer general stability from repeated tests involving only straightforward English-language comments.
How Should Researchers Measure Agreement Between Runs?
The appropriate measure depends on the type of output.
For single-label categorical classifications, researchers may calculate exact agreement or use suitable chance-corrected agreement statistics. For numerical outputs, they may examine absolute differences, variability, and agreement within predefined tolerances.
For qualitative interpretations, evaluation may require a documented rubric and independent human judgment rather than exact text matching.
| Output Type |
Possible Evaluation |
Important Limitation |
| Single-label categories |
Exact agreement, Cohen's kappa, confusion matrices |
Agreement does not establish correct labels |
| Multiple-label categories |
Label-specific agreement or set-based similarity |
Ordinary single-label measures may be inappropriate |
| Numerical values |
Absolute differences, variability, tolerance-based agreement |
Consistent values may still be incorrect |
| Free-text summaries |
Rubric-based semantic and factual evaluation |
Different wording may express equivalent findings |
| Analytical conclusions |
Sensitivity analysis across alternative outputs |
Stable conclusions do not establish valid measurements |
For categorical coding, stability testing requires attention to category-specific disagreement, not merely an overall percentage.
A model may agree consistently on common categories while frequently changing its decisions on uncommon but scientifically important observations.
Can Repeated AI Outputs Be Combined Through Majority Voting?
One possible procedure is to generate several classifications for each observation and select the most frequently assigned label.
This may produce a single final decision from variable outputs, but it should not be assumed to improve scientific accuracy.
Repeated generations from the same model may share systematic errors. If the model consistently misunderstands a coding rule, majority voting may simply reinforce that misunderstanding.
Researchers should define the aggregation rule before examining the final results and evaluate its performance against independent reference evidence.
They should also specify what happens when there is a tie or when no category receives sufficient support.
Majority voting is therefore an analytical design choice requiring validation, not a general solution to AI uncertainty.
Should Researchers Keep Rerunning Until the AI Gives a Consistent Answer?
Not without a predefined stopping rule.
Repeatedly generating outputs until the model produces a preferred or apparently stable response can introduce selective reporting.
For example, a researcher might stop after three consecutive classifications agree while ignoring several earlier disagreements. Reporting only the final matching responses would conceal the actual variability observed during the procedure.
A defensible protocol should specify the number of runs, stopping conditions, handling of invalid responses, and treatment of disagreements before the final analysis.
All outputs included in the evaluation should be retained where permitted, including failures and contradictory responses.
Watch Out
Do not treat repeated AI responses as independent confirmations of a factual claim. Multiple outputs from the same model may reflect the same underlying error. Independent verification requires evidence beyond the model's repeated assertions.
When Should Researchers Stop Testing and Reconsider the Method?
Repeated testing is useful only when it informs a methodological decision.
If outputs remain unstable despite clarified instructions and controlled settings, researchers should investigate whether the task is too ambiguous, the coding framework is inadequate, or the model is unsuitable for the intended measurement.
Additional executions may reveal more variation without resolving the underlying problem.
Researchers may need to revise the coding procedure, introduce independent adjudication, use a different analytical method, or reduce the role of AI in the study.
When the variation changes the substantive conclusions, the issue is no longer merely technical. It concerns whether the evidence is sufficiently dependable to support the research claim.
Preserve the Results of Every Consequential Test
Repeated-run testing creates analytical evidence that should be documented.
Relevant records include the evaluation dataset, prompt version, model identifier, settings, execution dates, outputs from each run, agreement calculations, and decisions made after examining disagreements.
Where AI outputs materially influence the findings, preserving the original responses supports independent examination of the stability analysis.
Researchers should distinguish exploratory pilot tests from the final evaluation and avoid presenting performance measured on repeatedly optimized development data as independent validation.
