03 · What You Need to Know
Why Poor Task Specification Produces Conflicting AI Research Outputs
What does it mean for a research task to be poorly specified?
A research task is poorly specified when the available instructions do not establish enough information to determine what the system should do or how its output should be evaluated.
For example, asking an AI model to classify research articles may be insufficient if the researcher does not define the classification categories, their boundaries, or how ambiguous cases should be handled.
Similarly, asking for the best statistical method may be underdetermined without specifying the research objective, outcome scale, data structure, and relevant assumptions.
Task specification concerns more than prompt wording. It includes the conceptual definitions, analytical objectives, decision procedures, and evidence needed to establish an appropriate result.
A grammatically clear question can still be methodologically incomplete.
Why can identical instructions support different interpretations?
Natural language frequently permits multiple reasonable interpretations.
Consider the instruction: "Identify the main theme of this interview."
One model may interpret main theme as the most frequently discussed topic. Another may identify the most theoretically significant pattern. A third may focus on the participant's most emotionally emphasized experience.
These outputs may differ because the researcher has not operationalized what main theme means within the analytical framework.
In qualitative research, the appropriate interpretation also depends on the epistemological orientation and analytical approach. There is no universally applicable rule that the most frequent topic must be the most meaningful theme.
Researchers should therefore define the intended analytical construct rather than assuming that familiar terminology carries one interpretation across models.
Which missing details commonly cause AI disagreement?
| Missing specification |
Example |
Possible disagreement |
| Research objective |
"Analyze the factors affecting performance." |
Models recommend causal, explanatory, or predictive methods. |
| Outcome definition |
"Compare student satisfaction." |
Models assume continuous, ordinal, or categorical outcomes. |
| Observation structure |
"Compare two groups." |
Models assume independent or paired observations. |
| Classification rules |
"Identify experimental studies." |
Models apply different definitions of experimental design. |
| Source boundaries |
"Extract the sample size." |
Models report recruited, completed, or analyzed participants. |
| Ambiguity handling |
"Assign one code to each excerpt." |
Models force different categories when evidence is insufficient. |
| Evaluation criteria |
"Choose the best model." |
Models prioritize accuracy, speed, cost, or interpretability differently. |
These examples illustrate why a task should be evaluated for conceptual and methodological completeness before attributing conflicting outputs solely to the AI systems.
What is the difference between task ambiguity and model unreliability?
Task ambiguity concerns insufficiently defined instructions or decision criteria. Model unreliability concerns failure to perform adequately under the relevant conditions.
The two can interact.
A model may behave inconsistently because the instructions permit multiple interpretations. However, it may also produce incorrect outputs despite receiving a clear and complete task specification.
Researchers should not assume that resolving ambiguity eliminates every source of error.
Task underspecification
Essential information needed to determine or evaluate the intended output has not been adequately established.
Model performance limitation
The system fails to satisfy the task's requirements despite an adequately defined and evaluable procedure.
Diagnosing the source of disagreement requires examining both the instructions and the model's observed behavior.
Why does the research objective matter?
Different research objectives require different analytical approaches.
For example, predicting which students may require additional academic support is not identical to estimating the causal effect of a teaching intervention.
A predictive model may prioritize out-of-sample performance, while a causal analysis requires explicit attention to identification assumptions and the interpretation of estimated effects.
When a prompt simply asks which model is best, AI systems may optimize for different implicit objectives.
Researchers should specify whether the task concerns description, prediction, explanation, causal inference, measurement, or another scientific purpose.
This clarification is necessary before evaluating the suitability of recommended methods.
How can missing statistical assumptions change AI advice?
Statistical methods depend on the structure of the data and the question being estimated.
For example, a comparison of independent groups differs from a comparison of repeated measurements on the same individuals.
Likewise, modeling a continuous outcome differs from modeling an ordered categorical outcome.
When these characteristics are omitted, AI models may silently assume different conditions and recommend different procedures.
Researchers should identify the relevant estimand, outcome scale, dependency structure, sampling design, and assumptions before accepting statistical advice.
The problem of identical prompts producing different methodological recommendations frequently involves these missing conditions.
Why does operationalization matter for AI-assisted qualitative research?
Qualitative concepts often require interpretation within a defined analytical tradition.
Terms such as theme, meaning, experience, engagement, and identity do not necessarily have one universal operational definition.
For example, thematic analysis may be conducted through approaches that differ in their assumptions about coding, researcher subjectivity, and theme development.
Braun and Clarke's work on reflexive thematic analysis emphasizes the interpretive and theoretically situated nature of qualitative analysis.
Researchers should therefore avoid assuming that an AI model can identify the correct theme without knowing the intended analytical approach.
Where AI assists qualitative analysis, its role should be explicitly defined and its outputs evaluated in relation to the researcher's methodological commitments.
Can clearer category definitions improve AI classification?
They may. Explicit category definitions can reduce avoidable differences in how models interpret a classification task.
For example, a research team classifying articles by study design can provide definitions distinguishing randomized experiments, nonrandomized interventions, observational studies, and qualitative inquiries.
Decision rules may also specify how to classify mixed-methods designs or studies containing multiple components.
However, category definitions should be grounded in the relevant methodology rather than invented solely to make AI outputs agree.
Researchers should validate the resulting procedure against independently established reference classifications.
Greater consistency does not automatically establish that the categories are conceptually appropriate or correctly applied.
What role should examples play in task specification?
Examples can clarify intended outputs, category boundaries, and decision rules.
For instance, a coding prompt may include examples of clearly eligible and ineligible studies, together with explanations of the criteria.
However, examples can also influence model behavior in ways that do not generalize to new data.
Researchers should distinguish examples used to develop the procedure from cases reserved for final evaluation.
Repeatedly modifying instructions based on performance on the same test cases can introduce overfitting and produce overly optimistic results.
The objective is to improve task clarity while preserving a credible assessment of generalization.
Should researchers specify what AI should do when information is missing?
Yes, when missing information affects the validity of the task.
A classification procedure may need an indeterminate category, an abstention option, or a rule requiring human review.
For example, an abstract may not contain enough information to determine whether participants were randomly assigned.
Forcing the model to choose between randomized and nonrandomized categories can produce unsupported classifications.
Researchers should specify whether the model may return insufficient information and how those cases will be handled.
However, permitting abstention does not guarantee that the model will recognize every uncertain case. Abstention behavior must also be evaluated.
How can researchers distinguish missing information from genuine uncertainty?
Missing information concerns details necessary to answer the question that have not been supplied or located.
Genuine scientific uncertainty concerns limitations in the available evidence or knowledge even after the question and relevant information have been appropriately specified.
For example, uncertainty about whether a study used paired observations may be resolved by consulting its methods section.
By contrast, uncertainty about an intervention's long-term effectiveness may persist because available studies provide insufficient evidence.
The distinction between AI disagreement and genuine scientific uncertainty requires researchers to establish what is unknown and why.
Can a task be specified too narrowly?
Yes. Excessively restrictive instructions may force outputs that do not adequately represent the evidence.
For example, requiring every qualitative excerpt to receive exactly one predefined code may conceal overlapping meanings or relevant ambiguity.
Similarly, requiring a single statistical recommendation without permitting discussion of assumptions may produce unwarranted certainty.
Good task specification should establish necessary constraints while allowing appropriate treatment of complexity and uncertainty.
The goal is not to eliminate every possible alternative response. It is to ensure that the outputs are meaningful and evaluable for the research purpose.
Does a structured output format improve reliability?
Structured outputs can make model responses easier to compare, process, and validate.
For example, researchers may require a classification label, supporting evidence, source location, and uncertainty status in separate fields.
This structure can reduce ambiguity about the expected format and facilitate automated checks.
However, a correctly formatted response may still contain an incorrect classification or fabricated supporting evidence.
Format compliance and substantive accuracy should be evaluated separately.
Researchers should not confuse machine-readable consistency with scientific validity.
How should researchers test whether revised instructions improve performance?
Researchers should establish a representative evaluation dataset with independently verified reference outcomes.
The original and revised procedures can then be compared using appropriate measures of accuracy, consistency, completeness, and consequential errors.
Where both procedures are applied to the same cases, paired comparisons may be appropriate.
Development cases used to refine instructions should be distinguished from independent test cases.
A reduction in disagreement between models is useful but does not establish improvement unless the resulting outputs are also evaluated for correctness.
Researchers should document changes to instructions and their observed effects.
How can disagreement rates be measured?
For categorical outputs, researchers can calculate the proportion of cases on which two models assign different labels.
Researchers may use additional agreement statistics where appropriate, particularly when category prevalence and chance agreement are relevant.
However, disagreement rates should not be interpreted as evidence of correctness. Two models can agree on the same incorrect label.
What if clearer instructions make both models agree on the wrong answer?
This is possible. A revised prompt may produce greater consistency by encouraging models to apply the same incorrect assumption.
For example, a classification rule may define quasi-experimental research incorrectly. Both models may then apply that rule consistently.
The observed disagreement rate could fall while classification validity deteriorates.
Researchers should therefore examine agreement and accuracy separately.
The possibility of shared errors among agreeing AI models illustrates why consistency alone is insufficient.
Can model updates complicate task specification?
Yes. A task specification that performs adequately with one model version may behave differently after an update.
Providers may change model behavior, system instructions, or inference configurations.
Researchers should preserve the task specification and relevant model identifiers when AI contributes to research measurements.
After a material model change, the procedure may require revalidation.
Clear instructions improve documentation but do not eliminate dependence on the computational system implementing them.
What should a complete AI research task specification contain?
The necessary details depend on the task, but a defensible specification generally identifies the scientific objective, intended inputs, relevant definitions, permitted outputs, decision rules, and treatment of missing or ambiguous information.
It should also establish how outputs will be evaluated and which decisions require independent verification.
For reproducibility, researchers may need to document the model version, prompts, preprocessing, decoding settings, and relevant execution environment.
The specification should be sufficiently detailed for another qualified researcher to understand the intended procedure.
However, it should not impose arbitrary constraints that undermine the conceptual validity of the analysis.
When should researchers stop revising the prompt?
Prompt revision should be guided by a defined methodological objective rather than continued until the model produces a preferred answer.
Researchers should avoid repeatedly modifying instructions using the final evaluation dataset.
Where performance remains inadequate despite reasonable clarification, the limitation may lie in the model's capabilities or the suitability of AI for the task.
The appropriate response may be to select another model, revise the research workflow, increase independent human review, or use a non-AI method.
Better prompting is not a universal remedy for inadequate model performance.
How should task specification be reported in a manuscript?
When AI contributes materially to research data or analysis, researchers should describe the task and its operational definitions sufficiently for readers to understand the procedure.
Relevant information may include the model, prompt or instruction template, category definitions, input materials, output rules, handling of ambiguous cases, and validation methods.
Extensive specifications may be placed in supplementary materials where appropriate.
Researchers should explain why the procedure is scientifically suitable, not merely report that a particular prompt produced consistent answers.
Documentation should distinguish the researcher's methodological decisions from the model-generated outputs.
