03 · What You Need to Know
Developing a Defensible Scoring Method for Partially Correct AI Responses
What makes an AI answer partially correct?
A partially correct response satisfies some of the task's requirements but fails to satisfy others.
This may occur because the answer contains a mixture of correct and incorrect claims, omits essential information, uses an inappropriate interpretation, or provides a correct conclusion supported by flawed reasoning.
For example, an AI response explaining thematic analysis might accurately describe coding and theme development while incorrectly presenting one methodological approach as mandatory for every thematic analysis study.
The answer contains useful information, but it also misrepresents an important methodological distinction.
Researchers should identify which components are correct and which are deficient before assigning an overall score.
Why is binary scoring sometimes insufficient?
Binary scoring assigns each response to a category such as correct or incorrect.
It can be appropriate when the task has a clearly defined outcome and partial correctness has little practical meaning.
For example, a system required to return one exact classification label may reasonably receive no credit for selecting the wrong category.
However, open-ended explanations often contain several independently assessable components.
Marking an answer entirely incorrect because it omits a minor qualification may discard useful information about the model's capabilities.
Conversely, marking it correct because its main conclusion is accurate may conceal serious errors in supporting claims.
Partial-credit scoring can provide additional information, but it requires explicit rules to prevent arbitrary judgments.
How is partial correctness different from incompleteness?
These concepts overlap but should not be treated as identical.
Partially correct response
An answer that satisfies some correctness requirements but fails others, potentially including incorrect claims or reasoning.
Incomplete response
An answer that omits information required by the task, even if everything it states is accurate.
For example, an AI response may correctly describe two assumptions of a statistical procedure while omitting a third. That is primarily a completeness problem.
Another response may list all three assumptions but describe one incorrectly. That involves factual inaccuracy.
Both may receive partial credit under some rubrics, but the nature of the deficiency should remain visible in the evaluation.
What scoring approaches can researchers use?
Several approaches are available, depending on the task and intended interpretation.
| Scoring approach |
How it works |
Appropriate use |
| Strict binary scoring |
Full credit only when all essential correctness criteria are satisfied |
Tasks where partial answers are unacceptable or consequential errors invalidate the result |
| Ordinal categories |
Responses classified as fully correct, partially correct, incorrect, or another defined category |
Studies requiring interpretable levels of correctness |
| Analytic rubric |
Separate criteria scored and combined using justified rules |
Complex explanations with multiple assessable components |
| Claim-level scoring |
Individual factual claims verified against evidence |
Long responses containing multiple factual assertions |
| Error-severity scoring |
Errors classified according to their consequences |
Tasks where some inaccuracies are substantially more serious than others |
These approaches can sometimes be combined.
For example, a study may use an analytic rubric to describe response quality while separately identifying whether the answer contains any critical error.
Researchers should avoid selecting the scoring method after examining which approach produces the most favorable model performance.
How should an analytic rubric be developed?
An analytic rubric divides the evaluation into criteria corresponding to the task's essential requirements.
For an AI-generated explanation of a statistical method, criteria might include correct identification of the method's purpose, accurate description of assumptions, appropriate interpretation of results, and recognition of relevant limitations.
Each criterion should have an operational definition and scoring guidance.
The rubric should distinguish what qualifies for full credit, partial credit, and no credit.
Where possible, researchers should pilot the rubric using responses outside the final evaluation dataset and revise ambiguous criteria before formal scoring.
The criteria should reflect the construct being evaluated rather than arbitrary preferences for response length or writing style.
Should every component receive equal weight?
Not necessarily.
Equal weighting is straightforward but may be inappropriate when some components are more consequential than others.
For example, correctly defining a statistical term may be less important than avoiding a fundamentally incorrect interpretation of the resulting analysis.
Researchers may assign different weights when they can justify the relative importance of the criteria.
However, weights should be established before final scoring whenever feasible.
Post hoc adjustments designed to improve a preferred model's score can introduce evaluation bias.
Researchers should also consider whether a weighted average is sufficient or whether certain critical errors should trigger a separate failure classification.
How can researchers calculate a partial-credit score?
One approach assigns points to predefined criteria and calculates the proportion of available points earned.
The resulting 55% represents performance under the specified weighted rubric.
It does not mean that exactly 55% of the response's factual claims are correct, nor does it establish that the answer is safe or acceptable for practical use.
In this example, the incorrect interpretation receives no credit despite the presence of accurate information elsewhere.
If incorrect interpretation is considered a critical failure, the study may need to report that failure separately rather than relying on the weighted score alone.
How should researchers identify critical errors?
A critical error is an error that substantially undermines the response's acceptability for the intended task.
Its definition depends on the application.
For example, a minor omission in a preliminary brainstorming response may be relatively inconsequential. An incorrect statement about a statistical assumption that leads to invalid analysis may be much more serious.
Researchers should define critical errors using task-relevant criteria and, where appropriate, disciplinary expertise.
A scoring procedure might report both a graded correctness score and the proportion of responses containing at least one critical error.
This prevents a high average score from concealing failures that would make particular outputs unsuitable for use.
How should claim-level correctness be evaluated?
Claim-level evaluation begins by identifying the substantive factual assertions in a response.
Each claim is then assessed against an appropriate reference standard.
For example, an AI-generated explanation of research ethics might contain separate claims about informed consent, confidentiality, ethics review, and data retention.
Researchers can classify each claim as supported, contradicted, unverifiable, or another justified category.
However, claim segmentation itself involves methodological choices. A long sentence may contain several independently verifiable assertions, and different evaluators may divide the text differently.
Researchers should therefore document the segmentation procedure and avoid assuming that all claims carry equal significance.
The distinction between claim-level and response-level correctness is central to interpreting accuracy in generative AI research.
What if the AI reaches the correct conclusion through incorrect reasoning?
A correct final answer does not necessarily establish that the explanation supporting it is valid.
For example, an AI system may recommend an appropriate statistical test while providing an incorrect justification for its selection.
If the research question concerns final-answer accuracy, the correct recommendation may receive credit under the defined scoring rule.
If the study evaluates methodological reasoning or explanation quality, the incorrect justification should affect the assessment.
Researchers should avoid inferring that a model used a particular internal reasoning process solely from the explanation it generated.
The observable explanation can be evaluated for logical coherence and evidentiary support, but it should not automatically be treated as a faithful account of the model's internal computation.
Researchers examining AI reasoning without access to internal reasoning should distinguish output-level justification from claims about internal mechanisms.
Should unsupported claims receive partial credit?
That depends on the scoring construct.
A claim that is plausible but unsupported by the available evidence should not automatically be treated as factually correct.
Researchers may distinguish unsupported assertions from demonstrably false statements, particularly when the evidence base is incomplete.
However, a response containing fabricated citations or invented empirical findings may be unacceptable even if other portions are accurate.
For research-related tasks, the presence of unsupported or fabricated claims may warrant a separate error category.
Where appropriate, researchers should examine hallucination rates as part of AI performance evaluation rather than allowing unsupported claims to disappear within an aggregate partial-credit score.
How can researchers make partial-credit scoring reliable?
Rubric-based scoring often requires judgment, particularly for open-ended responses.
Researchers should provide evaluators with explicit scoring instructions and examples illustrating the boundaries between categories.
Independent scoring by multiple evaluators may be useful when the judgments are subjective or consequential.
Agreement can be assessed using measures appropriate to the scoring scale, such as weighted kappa for ordinal categories or an intraclass correlation coefficient for certain numerical ratings.
Disagreements should be examined and resolved through a documented procedure when a final adjudicated score is required.
High inter-rater agreement supports consistency but does not automatically establish that the rubric measures the intended construct.
Should evaluators know which AI model produced an answer?
Where feasible, concealing model identity can reduce the influence of expectations about particular systems.
Evaluators may consciously or unconsciously assign different scores to responses they believe came from a highly regarded model.
Researchers can remove identifying labels and randomize the presentation order of responses.
However, complete blinding may be difficult when models have recognizable response styles.
Researchers should report the evaluation procedure and acknowledge relevant limitations.
These concerns become particularly important in studies that blind evaluators to whether responses came from AI or humans.
How should partial-credit results be reported?
Researchers should report the scoring rubric, criterion definitions, weights, and rules for handling critical errors.
Where appropriate, results may include the distribution of fully correct, partially correct, and incorrect responses alongside aggregate rubric scores.
Reporting only the average weighted score can conceal important differences in the nature of errors.
For example, two models may achieve the same average score while one frequently omits minor details and the other occasionally produces serious factual inaccuracies.
Researchers should interpret the scores in relation to the intended use and avoid describing a partial-credit percentage as conventional accuracy unless the operational definition supports that terminology.
