03 · What You Need to Know
Understanding and Reporting Zero-Shot and Few-Shot Research Conditions
What is zero-shot prompting?
Zero-shot prompting asks an AI model to perform a task without providing task-specific worked demonstrations in the prompt context.
For example, a researcher may instruct the model to classify a research abstract according to predefined methodological categories.
The prompt may contain operational definitions and decision rules, but it does not show completed examples of the same classification task.
Zero-shot prompting does not mean the model has never encountered similar tasks during training.
It refers to the absence of task-specific demonstrations in the supplied evaluation context.
Researchers should therefore avoid interpreting zero-shot performance as evidence of learning from no prior information whatsoever.
What is few-shot prompting?
Few-shot prompting supplies a small number of examples demonstrating how the intended task should be performed.
For example, a researcher may provide three educational research abstracts with their correct methodology labels before presenting a new abstract for classification.
These examples can communicate category boundaries, expected response formats, or patterns of task completion.
Few-shot prompting is often associated with in-context learning, where models use information supplied in the current context to guide responses without necessarily updating their underlying parameters.
However, the extent to which examples improve performance varies across models, tasks, and example configurations.
How do zero-shot and few-shot prompting differ?
| Feature |
Zero-shot prompting |
Few-shot prompting |
| Worked examples |
None supplied |
One or more supplied |
| Task instructions |
May include detailed instructions |
May include detailed instructions |
| Operational definitions |
May be provided |
May be provided |
| Context requirements |
Often shorter |
Usually longer because examples are included |
| Example-selection effects |
Not applicable to worked demonstrations |
May influence outcomes |
| Typical evaluation question |
How does the model perform without task-specific demonstrations? |
How does the model perform when supplied with selected demonstrations? |
| Reporting requirement |
Document instructions and relevant context |
Document instructions, examples, selection, and ordering |
The distinction concerns demonstrations rather than prompt length.
A lengthy zero-shot prompt may provide more substantive task guidance than a short few-shot prompt.
Is zero-shot prompting the same as giving no instructions?
No. Zero-shot prompts may contain extensive task instructions, category definitions, and output constraints.
For example, a researcher may provide detailed operational definitions of qualitative, quantitative, and mixed-methods research without including any worked classification examples.
That remains a zero-shot condition under the conventional distinction.
Researchers should therefore avoid assuming that zero-shot and few-shot comparisons isolate the effect of examples when other instructions also differ.
To investigate the contribution of demonstrations, the non-example portions of the prompts should generally be held constant where feasible.
Is one example considered few-shot prompting?
A prompt containing one worked example is often described more specifically as one-shot prompting.
Some discussions use few-shot as a broad category for prompts containing a small number of demonstrations, including one-shot configurations.
For scientific reporting, researchers should state the exact number of examples rather than relying solely on broad terminology.
For example, "three-shot prompting" communicates more information than simply describing the condition as few-shot.
Researchers should also specify whether examples are provided for every evaluation item or selected dynamically.
Why can few-shot examples influence AI performance?
Examples may communicate how a task should be interpreted and which outputs are expected.
For instance, a mixed-methods example may demonstrate that the category requires intentional integration of qualitative and quantitative approaches rather than merely the presence of numerical and textual information.
Such guidance may improve classification.
However, examples may also introduce misleading patterns or encourage inappropriate generalization.
Their influence depends on relevance, labels, ordering, and the relationship between demonstrations and evaluation items.
Researchers should treat example selection as a methodological decision.
How many examples should researchers provide?
There is no universally optimal number of demonstrations.
Performance may improve, remain unchanged, or decline as examples are added.
The appropriate number depends on task complexity, model behavior, available context, and the intended application.
Researchers may investigate several example counts during development.
However, selecting the count that performs best on the final evaluation dataset can introduce overfitting.
Development and confirmatory evaluation should be appropriately separated.
Should examples represent every classification category?
Category coverage may be important when examples are intended to clarify a classification framework.
For example, a study using three methodology categories might provide one demonstration for each category.
However, equal category representation is not automatically the best design.
Some studies may aim to reproduce realistic category prevalence, while others may intentionally balance demonstrations to isolate particular effects.
Researchers should justify the example distribution according to the research question.
They should avoid allowing unreported example imbalance to influence the interpretation of model performance.
How should few-shot examples be selected?
Examples may be selected randomly, purposively, or through a predefined retrieval or similarity procedure.
Each approach has methodological implications.
| Selection strategy |
Potential advantage |
Methodological concern |
| Random selection |
Reduces some discretionary selection |
May produce unrepresentative or weak demonstrations |
| Category-balanced selection |
Provides coverage of predefined categories |
May differ from real-world category prevalence |
| Expert-selected examples |
Can emphasize clear operational distinctions |
Introduces researcher discretion |
| Similarity-based retrieval |
Provides examples related to each test item |
May introduce leakage or inconsistent informational advantages |
| Fixed demonstration set |
Improves procedural standardization |
May favor particular task categories |
Researchers should report the selection strategy and relevant criteria rather than describing the examples merely as representative without supporting evidence.
Can few-shot example order affect results?
Yes. Research on in-context learning has demonstrated that demonstration order can influence model performance.
For example, placing a mixed-methods example immediately before a new abstract may affect the resulting classification differently from placing it first.
Researchers should document the order used during evaluation.
Where ordering effects are relevant, they may compare several predefined arrangements or randomize order according to a documented procedure.
The principles of prompt order effects are relevant to interpreting these differences.
Can few-shot examples contaminate an evaluation?
Yes. If examples contain evaluation items, near-duplicates, or reference answers that reveal the intended test outcomes, the resulting performance estimate may be compromised.
For example, supplying a previously classified abstract that is nearly identical to a test abstract may make the task substantially easier.
Researchers should establish separation between demonstration examples and held-out evaluation items.
However, semantic similarity is not automatically contamination. Similarity-based example retrieval may be an intended component of the system.
The study should define the permitted relationship between demonstrations and test items and interpret results accordingly.
Should zero-shot and few-shot prompts use identical instructions?
When the research question concerns the effect of adding demonstrations, the non-example instructions should generally remain equivalent.
For example, both conditions may use the same operational definitions and output requirements, with worked examples added only in the few-shot condition.
This helps isolate the contribution of demonstrations.
However, the addition of examples may also increase prompt length and computational cost.
Researchers should consider these consequences when interpreting the comparison.
If definitions or decision rules also change, the study should describe the conditions as broader prompting strategies rather than attributing every difference solely to the examples.
Can zero-shot prompting outperform few-shot prompting?
Yes. Few-shot prompting does not guarantee better performance.
Examples may be poorly selected, misleading, redundant, or inconsistent with the intended task.
Additional context may also affect model behavior in unexpected ways.
Researchers should evaluate the approaches empirically rather than assuming that more demonstrations necessarily improve results.
Findings should remain specific to the tested model, task, and prompt configurations.
How should zero-shot and few-shot performance be compared?
Researchers should establish a common evaluation dataset and reference standard.
Where possible, both conditions should use the same model configuration and task instructions, with the presence of demonstrations representing the planned difference.
Performance should be measured using metrics appropriate to the task.
For classification, these may include accuracy, precision, recall, and category-specific performance.
Repeated generations may be useful when response variability is relevant.
Statistical analysis should account for paired evaluation items and other dependencies.
How can the effect of few-shot prompting be calculated?
A descriptive comparison can express the difference in observed performance between few-shot and zero-shot conditions.
The difference indicates higher observed accuracy under the few-shot configuration.
It does not establish that every few-shot formulation will outperform every zero-shot formulation.
Researchers should report uncertainty and examine whether the result depends on the selected examples.
Should researchers compare multiple few-shot example sets?
When example selection may materially influence results, evaluating multiple sets can provide information about robustness.
For example, researchers may construct several category-balanced demonstration sets and assess performance across them.
However, repeated evaluations using the same test items introduce dependence that should be considered in the analysis.
Researchers should distinguish planned robustness analysis from selecting the best-performing example set after examining test results.
The latter can introduce selection bias.
These concerns connect few-shot evaluation with prompt cherry-picking.
Does few-shot prompting involve model training?
Ordinarily, few-shot prompting does not update the model's underlying parameters.
Instead, examples are supplied within the model's input context to influence its immediate behavior.
This differs from fine-tuning, which involves modifying model parameters through a training procedure.
However, researchers should not infer the complete internal mechanism of a proprietary model solely from its observed behavior.
For methodological reporting, the important distinction is whether examples were supplied during inference or used in a separate parameter-updating process.
Can few-shot prompting increase evaluation cost?
Yes. Worked examples increase the amount of input material processed by the model.
This may increase token consumption, processing time, and monetary cost, depending on the system.
For large evaluation datasets, these differences may become substantial.
Researchers should consider whether the performance benefit justifies the additional resources when practical efficiency is part of the research question.
The principles of AI cost and speed evaluation are relevant.
Should few-shot examples be included in supplementary materials?
Generally, yes, when they materially influence the evaluation procedure.
Researchers should preserve the exact examples, labels, ordering, and instructions supplied to the model.
If examples are selected dynamically, the selection algorithm and relevant execution records may also be necessary.
However, examples may contain confidential research materials or participant information.
Researchers should apply appropriate safeguards and provide redacted or restricted-access records when required.
Public transparency should not override legitimate data-protection obligations.
How should zero-shot and few-shot prompting be reported in a paper?
The methods section should identify the prompting condition, model configuration, task instructions, and evaluation procedure.
For few-shot prompting, researchers should additionally specify the number and selection of demonstrations.
Illustrative Methods Description
"The study compared zero-shot and three-shot prompting for classification of educational research abstracts. Both conditions used identical operational definitions, output requirements, and model settings. The three-shot condition additionally included one labeled demonstration for each methodology category, selected from a development dataset excluded from formal evaluation. Demonstration order was fixed and documented. The complete prompts, examples, and evaluation procedures are provided in the supplementary materials."
This description identifies the principal difference between conditions without implying that the observed result generalizes to every possible demonstration set.
