Should AI Prompts Be Prespecified in a Research Study?

Should Researchers Finalize Their AI Prompts Before Collecting Research Results?

You are evaluating whether a generative AI model can identify methodological weaknesses in research proposals. You develop an initial prompt, test it on several proposals, and discover that its recommendations are inconsistent.

You revise the wording, add methodological definitions, and introduce examples. Performance improves.

After examining the formal evaluation results, however, you notice that the model performs poorly on mixed-methods proposals. You modify the prompt again, rerun the evaluation, and obtain a higher accuracy score.

Which result should be reported? Is the final prompt a legitimate improvement, or has the evaluation become influenced by the results it was supposed to measure independently?

This problem resembles a familiar methodological concern: analytical decisions made after observing outcomes can introduce selection bias when presented as though they were determined in advance.

Prespecifying prompts may help distinguish legitimate development from confirmatory evaluation, but researchers need to understand when such prespecification is necessary and what flexibility remains appropriate.

Prespecification Is Particularly Important for Confirmatory AI Research

Prespecifying Prompts Without Preventing Legitimate Methodological Development

What does it mean to prespecify an AI prompt?

Prompt prespecification means establishing and documenting the prompt or prompt-selection procedure before the relevant confirmatory evaluation outcomes are examined.

For example, a researcher may finalize a prompt instructing AI to classify research abstracts according to predefined methodological categories.

The researcher then applies that prompt to a held-out evaluation dataset without modifying it in response to the observed test results.

Prespecification may concern the exact prompt text, a reusable template, a predefined set of prompt variants, or a documented optimization algorithm.

The appropriate form depends on the research design.

The central requirement is that the confirmatory procedure is established independently of the outcomes used to evaluate it.

Why is prompt prespecification important?

Generative AI performance can vary across prompt formulations.

Researchers who test numerous prompts and report only the most successful one may obtain an optimistic estimate of performance.

This concern is particularly important when the same evaluation dataset is used repeatedly to guide prompt revisions.

Prespecification can reduce opportunities for such selective optimization.

It also helps readers distinguish the originally intended procedure from decisions made after observing results.

However, prespecification does not guarantee that a prompt is valid, unbiased, or scientifically appropriate. It improves transparency about the procedure rather than automatically establishing its quality.

What is the difference between prespecification and preregistration?

A prompt may be prespecified in a dated institutional protocol without being publicly preregistered.

Conversely, a preregistered AI study may include its exact prompts, prompt-selection procedures, or links to version-controlled supplementary materials.

Preregistration provides a stronger external record of timing when the registration is appropriately preserved.

However, a public registration alone does not establish that the selected procedure is methodologically sound.

Which AI studies benefit most from prompt prespecification?

Prespecification should be proportionate to the role prompting plays in the study.

It is particularly valuable when prompt choices could affect the primary outcome or the interpretation of a confirmatory hypothesis.

Must the exact prompt be finalized before any testing?

No. Pilot testing is often necessary to develop a usable research procedure.

Researchers may discover that an initial prompt contains ambiguous instructions, produces invalid output formats, or fails to distinguish important methodological categories.

Revising the prompt during development is not inherently problematic.

The important distinction is between development data and the independent data used for final evaluation.

Researchers should avoid presenting performance measured on repeatedly consulted development data as though it were an unbiased estimate from a held-out test.

The principles of pilot-testing AI prompts are therefore compatible with appropriate prespecification.

What is the difference between exploratory and confirmatory prompting?

Exploratory prompting investigates possible instructions, strategies, and model behaviors without necessarily testing a fixed hypothesis.

Confirmatory prompting applies a predefined procedure to evaluate a specified research question or hypothesis.

Both approaches can make legitimate scientific contributions.

However, exploratory findings should not be presented as though the prompt conditions and hypotheses were fixed independently of the observed outcomes.

Researchers should distinguish the two stages in their methodology and reporting.

This distinction is particularly important when many prompt variants have been examined.

How can researchers separate prompt development from final evaluation?

One approach uses separate development and evaluation datasets.

Researchers develop and refine prompts using the development data, then finalize the procedure before applying it to the held-out evaluation set.

For example, 50 research abstracts may be reserved for prompt development, while a separate set of 200 abstracts is used for confirmatory testing.

However, simply dividing a dataset does not guarantee independence if near-duplicate items, related documents, or shared information appear across the two partitions.

Researchers should consider contamination, representativeness, and appropriate data separation.

When the available dataset is small, alternative designs such as nested resampling may be appropriate, provided the complete selection procedure is respected.

What happens when researchers optimize prompts on the final test set?

Repeatedly inspecting final test results and modifying prompts accordingly can introduce evaluation overfitting.

For example, a researcher may test 20 prompts on the same 100 abstracts and select the one achieving the highest accuracy.

The selected score may partly reflect favorable variation specific to those abstracts.

Reporting that score as an independent estimate of general performance may therefore be misleading.

The problem becomes more pronounced when researchers continue modifying prompts until a desired threshold is reached.

Researchers should document such development and obtain fresh independent evaluation evidence when making confirmatory performance claims.

Can selecting the best prompt create statistical optimism?

Yes. Even if several prompts have similar underlying performance, their observed scores may differ because of evaluation noise.

Selecting the highest observed score can produce an upwardly biased estimate of the selected prompt's generalization performance.

The selected prompt should be evaluated on data not used to make the selection decision when the study seeks an independent estimate.

Researchers should also avoid treating the selected prompt as universally optimal across tasks or model versions.

Should researchers prespecify one prompt or several prompts?

Either approach may be appropriate.

A study evaluating one deployed AI workflow may use a single finalized prompt.

A study investigating prompt sensitivity may prespecify several prompt variants and compare their outcomes.

Another study may define a prompt-selection procedure that chooses among candidates using development data.

The relevant requirement is that the primary evaluation procedure and comparisons are defined before the confirmatory outcomes are examined.

Researchers should avoid selecting which prompt comparisons to report solely because they produce favorable findings.

Can adaptive prompting be prespecified?

Yes. Prespecification does not necessarily require one immutable prompt.

An adaptive procedure may modify prompts according to predefined rules.

For example, a system may request additional information when an initial classification is uncertain or use a second prompt when the first response fails a validation check.

Researchers can prespecify the conditions triggering these actions, the permitted prompt variants, and the rules for selecting final outputs.

The complete procedure then becomes the object of evaluation.

However, researchers should distinguish predefined adaptation from discretionary changes introduced after examining confirmatory outcomes.

What should a prespecified prompt protocol contain?

A prompt protocol should provide enough information to establish how the AI-assisted procedure will operate.

The level of detail should reflect the complexity of the study and the extent to which prompt decisions influence its conclusions.

Should the AI model version also be prespecified?

When feasible, researchers should identify the intended model and configuration before confirmatory testing.

However, commercial AI providers may update systems or retire model versions without advance notice.

Researchers should document the model actually used, the testing period, and any material changes.

If a model update occurs during the study, the researchers should assess whether the evaluation conditions remain comparable.

Prespecification cannot guarantee that an external service will remain unchanged.

It can, however, establish how the study intends to handle such changes.

What if a prespecified prompt contains an error?

Prespecification does not require researchers to continue using a demonstrably defective procedure.

For example, a prompt may contain an incorrect operational definition or a formatting instruction that prevents valid data collection.

Researchers may need to correct the problem.

However, the amendment should be documented, including when the error was discovered, what changed, and which evaluation outcomes had already been examined.

If the change was informed by final test performance, a fresh confirmatory evaluation may be necessary.

The original procedure should not be silently replaced in the research record.

How should prompt amendments be documented?

Researchers should preserve the original prompt and each material revision.

A change log may record the date, reason, affected instruction, and implications for previously collected results.

This hypothetical record illustrates how version history can distinguish development decisions from post-evaluation changes.

Researchers should avoid describing an amended prompt as originally prespecified when it was not.

Does preregistration prevent prompt cherry-picking?

Preregistration can reduce opportunities for undisclosed selective reporting by establishing a time-stamped plan.

However, it does not automatically prevent deviations, hidden exploratory analyses, or inappropriate interpretation.

Researchers may still conduct additional prompt experiments after registration.

The important requirement is to disclose those analyses and distinguish them from the original confirmatory plan.

The methodological concerns associated with prompt cherry-picking remain relevant even in preregistered studies.

Can researchers preregister a prompt-selection algorithm?

Yes. Rather than specifying one fixed prompt, researchers may preregister a reproducible selection procedure.

For example, the protocol may define a set of candidate prompts, a development dataset, an accuracy metric, and a rule for selecting the highest-performing candidate.

The procedure can then be applied without consulting the final evaluation outcomes.

However, the candidate-generation process and any manual intervention should also be described.

A supposedly automated selection procedure may still contain discretionary decisions that influence the result.

The study should preserve enough information to reconstruct the selection process where feasible.

Should researchers prespecify prompt sensitivity analysis?

When robustness to prompt wording is central to the research question, prespecifying sensitivity analyses can strengthen the study.

For example, researchers may define five semantically comparable prompt variants and examine how classification accuracy changes.

However, semantically comparable wording should be distinguished from prompts that supply different examples or substantive instructions.

Researchers should identify which variations are intended to test robustness and which represent different experimental conditions.

Unplanned sensitivity analyses may still be useful, but they should be reported as exploratory.

Should all prompt variants be archived?

Preserving relevant prompt variants supports transparency about the development and evaluation process.

Researchers may maintain version-controlled records containing prompt text, selection decisions, and associated development results.

However, not every informal brainstorming exchange needs to appear in the main manuscript.

The appropriate level of archival detail depends on whether a prompt variant materially influenced the procedure or conclusions.

Confidential or sensitive information should be protected.

Researchers should distinguish the archival record from the smaller set of materials necessary for public reporting.

How does prespecification relate to research integrity?

Prespecification can help make analytical decisions more transparent and reduce undisclosed outcome-driven changes.

However, failing to prespecify a prompt is not automatically research misconduct.

Exploratory studies may legitimately develop prompts iteratively.

The research-integrity concern becomes more serious when researchers conceal material changes, selectively report favorable results, or falsely describe exploratory decisions as prespecified.

Researchers should distinguish methodological limitations from deliberate misrepresentation.

Transparency about the sequence of decisions is essential.

How should prespecified prompting be reported?

The methods section should identify when the prompt was finalized, how it was developed, whether the study was preregistered, and whether deviations occurred.

For example:

If deviations occurred, the manuscript should describe them and distinguish the original analysis from subsequent exploratory work.

Such reporting allows readers to understand whether the reported performance reflects an independently evaluated procedure or one selected using the evaluation outcomes.

Prespecifying a Prompt Before Evaluating Research Classification Accuracy

The example illustrates that prespecification does not prohibit prompt development.

Instead, it establishes a boundary between the process used to select a procedure and the independent evidence used to evaluate that procedure.

Common Misconceptions About Prespecified AI Prompts

Deciding When and How to Prespecify AI Prompts

Researchers should determine whether their study makes confirmatory claims and whether prompt selection could influence the reported outcomes.

Prespecification should not become an exercise in administrative rigidity.

Its scientific value lies in establishing which decisions preceded the evidence and which were informed by it.

Before Finalizing a Prespecified Prompt Protocol

Frequently Asked Questions About Prespecifying AI Prompts

Prespecification Separates Prompt Development From Independent Evaluation

How to Cite This Guide

This guide is intended to be read, shared, and used in research, teaching, and academic work. If you draw on its ideas, explanations, or other content, please acknowledge the source by citing the guide. Doing so gives appropriate credit and helps your readers locate the original resource.