03 · What You Need to Know
Evaluating AI Classification Accuracy and Qualitative Coding Quality
What is the difference between AI classification and qualitative coding?
Classification assigns observations to defined categories. In a conventional supervised classification task, the categories are established in advance, and the system's predictions can be compared with reference labels.
For example, an AI model may classify research abstracts as qualitative, quantitative, or mixed methods.
Qualitative coding involves assigning labels to segments of text or other material to represent concepts, meanings, experiences, or analytical observations.
Some qualitative coding procedures use predefined codebooks and resemble classification. Others involve iterative interpretation, reflexivity, and development of codes during analysis.
Predefined classification
Assigning items to established categories using explicit decision rules, often with reference labels against which predictions can be evaluated.
Interpretive qualitative coding
Assigning or developing codes to represent meanings and patterns within data, potentially involving contextual interpretation and evolving analytical decisions.
The distinction matters because conventional accuracy metrics assume a meaningful basis for identifying correct predictions.
In interpretive qualitative research, disagreement may reflect different but defensible readings of the data rather than a simple classification error.
Why must the evaluation begin with a coding framework?
Before evaluating AI-generated labels, researchers should define what the labels are intended to represent.
For a classification task, this usually involves specifying the available categories, their definitions, and the rules governing ambiguous cases.
For deductive qualitative coding, the codebook may include code names, conceptual definitions, inclusion and exclusion criteria, and examples.
Without these definitions, an AI system may assign labels that appear plausible but do not correspond to the intended constructs.
Researchers should also specify the unit of analysis. Coding a complete interview response is not necessarily equivalent to coding individual sentences or meaning units.
Differences in segmentation can produce disagreement even when the substantive interpretations are similar.
What should count as the reference standard?
For predefined classification tasks, researchers may establish reference labels through verified source information or independently adjudicated expert judgments.
For example, research methodology classifications may be established through review of full papers using explicit methodological criteria.
For qualitative coding, the reference standard depends on the analytical approach.
In codebook-based content analysis, an independently developed coding scheme and adjudicated human labels may provide a suitable comparison.
In reflexive thematic analysis, however, treating one researcher's interpretation as the uniquely correct answer may be inconsistent with the methodology's assumptions.
Researchers should distinguish agreement with a particular coding framework from proof that an interpretation is universally correct.
The broader principles of establishing a defensible AI evaluation reference standard apply, but the form of that standard must fit the analytical approach.
Which metrics are appropriate for single-label classification?
When every item belongs to one predefined category, several conventional classification measures may be useful.
| Metric |
What it measures |
Important consideration |
| Accuracy |
Proportion of predictions matching reference labels |
May conceal poor performance on minority categories |
| Precision |
Proportion of predictions assigned to a category that are correct |
Useful when false-positive assignments matter |
| Recall |
Proportion of actual category members correctly identified |
Useful when missed instances matter |
| F1 score |
Harmonic mean of precision and recall |
Does not include true negatives in its calculation |
| Confusion matrix |
Distribution of predicted categories against reference categories |
Reveals which categories are confused |
Researchers should select metrics according to the consequences of different classification errors.
For example, a literature screening system may require high recall when missing relevant studies would undermine the completeness of a review.
By contrast, another application may prioritize precision when false-positive classifications create substantial additional work.
How are precision, recall, and F1 calculated?
For a particular category treated as positive, precision and recall are calculated from the numbers of true positives, false positives, and false negatives.
The F1 score is approximately 71.1% when expressed as a percentage.
It summarizes the balance between precision and recall for the specified category. It does not establish the quality of the broader qualitative interpretation or explain why particular excerpts were misclassified.
For multiclass classification, researchers should specify whether metrics are calculated separately by category or aggregated using macro, micro, or weighted averaging.
What changes when one text segment can receive multiple codes?
Qualitative material may legitimately receive several codes.
For example, a student's interview response about online learning might express technical frustration, reduced motivation, and limited access to learning resources.
A single-label classification procedure would be inappropriate if the coding framework permits all three codes to apply.
Researchers may instead evaluate each code as a separate binary decision or use multilabel measures such as example-based F1, Hamming loss, or exact-set match.
These measures answer different questions.
Exact-set match requires the complete predicted set of labels to match the reference set. It can be demanding when many labels are possible.
Per-code precision and recall provide more detailed information about which concepts the AI identifies or overlooks.
Researchers should report the selected metric's meaning and avoid presenting multilabel performance as ordinary single-label accuracy.
Can Cohen's kappa measure agreement between AI and human coding?
Cohen's kappa can be used to quantify agreement between two sets of categorical labels under appropriate conditions.
It adjusts observed agreement for the level expected under a model based on the marginal category distributions.
A kappa of 0.70 indicates agreement beyond the expected level under the measure's assumptions.
It does not mean that the AI is 70% accurate or that the human coder's interpretations are objectively correct.
Kappa can also be affected by category prevalence and marginal distributions.
Researchers should choose agreement measures appropriate to the coding structure and avoid treating a particular coefficient as universally sufficient evidence of validity.
Does agreement with human coders establish coding validity?
Not necessarily.
Agreement indicates consistency between coding decisions. Validity concerns whether those decisions adequately represent the intended constructs or meanings.
An AI model may reproduce a human coder's labels consistently while both apply a flawed codebook.
Conversely, an AI-generated code may differ from a human label while identifying a defensible interpretation that deserves further consideration.
Researchers should examine disagreements qualitatively rather than assuming every deviation from the reference labels is equally informative.
In interpretive qualitative research, analytical credibility may also depend on contextual engagement, reflexivity, transparency of interpretation, and coherence with the methodological approach.
Should inter-coder reliability always be calculated in qualitative research?
No. The appropriateness of inter-coder reliability depends on the qualitative methodology.
In some forms of structured content analysis or codebook-based thematic analysis, agreement statistics may help evaluate consistency in applying predefined categories.
In reflexive thematic analysis, however, coding is understood as an interpretive activity rather than a process of discovering one objectively correct label for every segment.
Braun and Clarke have argued that conventional coding-reliability expectations are not necessarily compatible with reflexive thematic analysis.
Researchers should therefore avoid imposing agreement coefficients on qualitative approaches whose epistemological assumptions do not support that form of validation.
If AI is used to generate candidate codes within an interpretive approach, its contributions may be evaluated through analytical usefulness and critical examination rather than simple agreement percentages.
How should researchers evaluate AI-generated qualitative codes?
For interpretive coding, researchers can examine whether proposed codes are grounded in the data, conceptually coherent, appropriately specific, and useful for the research question.
A code should not be considered valid merely because its label sounds plausible.
Researchers should inspect the excerpts assigned to each code and determine whether the coding decision is supported by the surrounding context.
They should also examine whether the AI overlooks implicit meanings, imposes stereotypical interpretations, or creates categories not supported by the data.
AI-generated codes may serve as provisional analytical suggestions, but their acceptance should follow the methodological commitments of the study.
When AI-generated responses are used to test a coding procedure, evaluating a coding framework using synthetic responses requires additional care because simulated material is not equivalent to empirical participant data.
How should ambiguous or disputed coding decisions be handled?
Ambiguity should be addressed through documented rules appropriate to the task.
For predefined classification, researchers may consult additional source material, adjudicate disagreements, or permit an uncertainty category.
For interpretive qualitative coding, disagreement may warrant further analytical discussion rather than forced consensus.
Researchers should distinguish genuinely ambiguous source material from errors caused by an inadequate code definition.
Excluding ambiguous items without justification can make classification performance appear stronger than it is.
Where uncertainty remains consequential, sensitivity analyses may show how alternative defensible coding decisions affect the results.
How should researchers examine classification errors?
Aggregate accuracy can conceal systematic patterns.
For example, an AI model may classify straightforward quantitative abstracts correctly while frequently misidentifying mixed-methods studies.
A confusion matrix can reveal whether particular categories are consistently confused.
Researchers should also inspect errors associated with ambiguous wording, document length, language, and other relevant characteristics.
For qualitative coding, error analysis may examine missed meanings, inappropriate code assignments, excessive generalization, or failures to recognize contextual nuance.
The consequences of different errors should inform interpretation rather than being reduced automatically to one average score.
Should researchers repeat the same coding task?
Repeated testing may be useful when generative AI produces different codes or classifications from identical inputs.
For predefined classification, researchers can examine whether labels remain stable across repeated generations.
For qualitative coding, repeated outputs may reveal alternative interpretations, but variation should not automatically be treated as evidence of invalidity.
Researchers should distinguish consistency within the same prompt from changes caused by prompt wording or codebook modifications.
The evaluation should document the number of generations, the model settings, and the procedure used to aggregate or compare outputs.
Where relevant, response stability should be assessed as a separate performance dimension.
Can another AI system evaluate AI-generated codes?
An AI-based evaluator may help compare labels, identify inconsistencies, or apply a scoring rubric.
However, the evaluator may reproduce similar biases or misinterpret the coding framework.
Researchers should validate automated judgments against an appropriate independent reference procedure.
For qualitative coding, AI evaluation should not substitute for methodological interpretation when the study requires contextual or reflexive engagement with the data.
Any use of model-based grading should be documented, including the evaluator model, prompts, scoring criteria, and validation evidence.
How should coding and classification results be reported?
The report should identify the task, dataset, unit of analysis, coding framework, reference standard, model configuration, prompts, and evaluation measures.
For classification, researchers should report relevant category-specific results and the treatment of invalid or ambiguous outputs.
For qualitative coding, they should explain the analytical approach and why the selected evaluation criteria are methodologically appropriate.
Where feasible, researchers can preserve codebooks, anonymized coding records, adjudication decisions, and analysis scripts.
Sharing participant material must remain consistent with consent, confidentiality, ethics approvals, and applicable data-protection requirements.
The broader principles of AI performance evaluation remain relevant, but the interpretation of coding quality must reflect the nature of the task.
