Can Generative AI Apply a Qualitative Research Codebook?

Can You Give AI a Codebook and Ask It to Code Your Interviews?

You have developed a qualitative research codebook containing labels, definitions, and examples. Perhaps your research team has already tested it on several interview transcripts and agreed on how the codes should be applied. You now have dozens of additional transcripts awaiting analysis.

Could you give the codebook to ChatGPT or another generative AI system, provide the transcripts, and instruct it to apply the same coding rules?

The task seems straightforward because the codes have already been defined. Unlike generating entirely new codes, the system would be selecting from an established set of categories.

However, applying a codebook involves more than recognizing keywords. The model must interpret the meaning of each passage, distinguish similar codes, respect exclusion rules, and recognize cases that do not fit the framework. An apparently consistent coding table can conceal systematic misclassification.

Yes, AI Can Apply a Codebook, but Its Assignments Require Validation

How AI Applies a Qualitative Codebook and Where Errors Arise

What Does It Mean to Apply a Qualitative Codebook?

A qualitative codebook is a structured reference that defines the codes used to classify or organize qualitative material. It may contain code names, definitions, inclusion criteria, exclusion criteria, examples, and relationships among codes.

Applying a codebook means examining a passage and determining whether its content satisfies the criteria for one or more predefined codes.

For example, a study examining barriers to online learning might use the following codes:

• Connectivity Barrier: Participation is disrupted by internet access, network stability, or connection quality.
• Device Access Barrier: Participation is limited by the availability, suitability, or sharing of digital devices.
• Household Environment Barrier: Participation is affected by noise, interruptions, lack of space, or household activities.
• Participation Anxiety: The participant describes apprehension about speaking, being evaluated, or interacting during class.

AI could examine an interview excerpt and select the code or codes that appear applicable.

This differs from asking AI to generate initial qualitative codes, where the analytical categories themselves are still being developed.

What Should a Codebook Contain Before AI Uses It?

A list of code names is usually insufficient for dependable classification. Labels such as "Access," "Engagement," or "Support" may be interpreted differently unless their boundaries are specified.

Methodological guidance from the US Centers for Disease Control and Prevention describes the value of codebooks containing definitions, examples, and guidance concerning when codes should or should not be applied. These features support more consistent coding within research teams. CDC guidance on qualitative analytical rigor discusses these practices in detail.

A practical codebook for AI-assisted classification should generally specify:

The examples are hypothetical. Actual codebooks should be developed in relation to the study's research questions, theoretical framework, and data.

Clear rules make AI assignments easier to evaluate, but they do not eliminate the need for interpretation.

How Does AI Decide Which Code to Apply?

When provided with a codebook and a passage, a generative AI model produces an assignment based on the instructions, code definitions, examples, and text available in its context.

It does not necessarily follow the codebook as a deterministic rule-based program would. A model may interpret a definition broadly, prioritize a familiar phrase, or infer an unstated meaning.

Consider the statement:

"I couldn't join the discussion because my brother was using our only laptop."

The appropriate assignment under the illustrative codebook is DEVICE_ACCESS_BARRIER.

However, a model might assign CONNECTIVITY_BARRIER because the statement concerns difficulty joining an online activity. That classification would confuse access to a device with access to an internet connection.

The distinction may appear minor, but repeated misclassification can distort the study's findings, particularly when researchers compare the frequency or distribution of different barriers.

Why Inclusion and Exclusion Criteria Matter

Definitions explain what a code represents. Inclusion and exclusion criteria clarify its boundaries.

Suppose the code PARTICIPATION_ANXIETY applies when participants explicitly describe fear, embarrassment, apprehension, or concern about being judged during classroom interaction.

Now consider:

"I usually keep my microphone off during discussions."

Without further context, the statement does not establish anxiety. The participant might be experiencing background noise, technological limitations, personal preference, or a classroom rule.

An AI system that automatically assigns PARTICIPATION_ANXIETY would introduce an unsupported inference.

An exclusion rule could specify that microphone avoidance alone is insufficient unless the participant describes anxiety or another qualifying indication.

Such boundaries are particularly useful when multiple codes describe related but distinct experiences.

Can AI Apply Multiple Codes to the Same Passage?

Yes, provided the codebook permits multiple assignments.

Consider:

"I was nervous about speaking, and when I finally tried, my internet connection dropped."

Under the illustrative definitions, the passage supports both PARTICIPATION_ANXIETY and CONNECTIVITY_BARRIER.

If AI is instructed to assign exactly one code, it may discard relevant information. Conversely, if every remotely related code is allowed, the system may overcode passages.

The codebook should specify whether codes are mutually exclusive, whether multiple coding is permitted, and how overlapping concepts should be handled.

These decisions should follow the analytical purpose rather than the technical convenience of the AI system.

What Should AI Do When No Code Fits?

A well-designed coding workflow should allow the possibility that no existing code applies.

Suppose a participant says:

"I prefer studying early in the morning because I feel more productive."

If the codebook contains only the four barriers described earlier, none may apply.

Forcing the model to select the closest available code can create false classifications. The system should be permitted to return an explicit no-code result when the passage does not satisfy any definition.

It may also be useful to flag passages that appear relevant to the research question but fall outside the current coding framework. Researchers can then decide whether the codebook requires revision.

This is particularly important when a deductive coding framework encounters unexpected experiences in the data.

Can AI Handle Ambiguous or Overlapping Codes?

AI can compare code definitions and suggest which category is more appropriate, but ambiguous passages remain challenging.

For example, a participant may say, "I stopped joining online classes because I couldn't keep up."

The statement could concern connectivity, academic workload, instructional pace, or personal circumstances. Without additional context, assigning a specific barrier code would be speculative.

Rather than forcing a confident classification, the coding procedure should allow an uncertainty flag and a brief explanation of what information is missing.

Researchers should also examine whether ambiguity results from insufficient data or poorly distinguished code definitions.

When multiple codes repeatedly overlap, revising the codebook may be more appropriate than repeatedly asking AI to choose between them.

Can AI Apply a Codebook Consistently Across Hundreds of Interviews?

Potentially, but consistency should be demonstrated rather than assumed.

Large-scale coding may involve dividing transcripts into batches, processing different participants separately, or using automated workflows. These procedures introduce opportunities for inconsistency.

For example, the model might apply a code narrowly in one batch and more broadly in another. Changes to prompts, model versions, examples, or surrounding context may also influence assignments.

This phenomenon is often described as coding drift when the operational meaning or application of codes changes during analysis.

Researchers can reduce avoidable variation by maintaining a stable codebook version, using consistent instructions, preserving transcript context, and periodically examining coding decisions against a reference set.

Nevertheless, identical prompts do not necessarily guarantee identical outputs across systems or repeated runs.

How Should AI Coding Accuracy Be Evaluated?

Evaluation should match the coding task and methodological approach.

For structured classification, researchers may compare AI assignments with independently reviewed coding decisions. A validation set should include clear examples, ambiguous passages, overlapping codes, and statements that should receive no code.

Several measures may be useful:

• Agreement: How often AI and reference coding decisions coincide.
• Precision: Among passages assigned a particular code, the proportion that genuinely satisfy its criteria under the reference standard.
• Recall: Among passages that satisfy a code's criteria, the proportion correctly identified by AI.
• Code-specific error patterns: Which categories are most frequently confused or omitted.
• Boundary-case performance: How the system handles ambiguous excerpts, overlapping categories, and no-code cases.

These measures require a clearly specified evaluation procedure. In multi-label coding, where several codes may apply to one passage, researchers should select metrics appropriate to that structure rather than relying on a single overall accuracy percentage.

Agreement coefficients such as Cohen's kappa may also be appropriate for certain designs, although their interpretation depends on the coding structure and prevalence of categories.

Importantly, human coding is not automatically an error-free gold standard. Reference assignments may require expert review and adjudication, and disagreements can reveal ambiguity in the coding framework itself.

What Does Methodological Research Recommend?

The CDC's guidance on collecting and analyzing qualitative data emphasizes repeated engagement with transcripts and the use of written code definitions and rules to reduce changes in coding practice. Its guidance on analytical rigor also discusses codebooks, examples, and collaborative refinement. The CDC Field Epidemiology Manual provides an accessible methodological reference.

A published case study on codebook development demonstrates how definitions, qualifications, exclusions, and examples can be documented to support systematic analysis. The approach is especially relevant when researchers need to explain why particular passages received particular codes.

These methodological principles provide a foundation for evaluating AI-assisted coding. They should not be interpreted as proof that a particular language model can apply any codebook accurately.

Does Codebook Application Fit Every Qualitative Methodology?

No. The appropriateness of a fixed coding framework depends on the analytical tradition.

Codebooks are commonly used in qualitative content analysis, framework analysis, and some forms of thematic analysis. They may support consistent classification when the research design calls for predefined or progressively refined categories.

However, Braun and Clarke distinguish codebook approaches from reflexive thematic analysis, which foregrounds the researcher's interpretive engagement and does not treat coding agreement as the principal measure of analytical quality.

Using AI to enforce a fixed codebook may therefore be compatible with one analytical approach but inconsistent with another if it substitutes classification reliability for the intended interpretive process.

Researchers should examine whether AI-assisted coding fits the epistemological assumptions of their study before selecting a workflow.

What About Confidentiality and Participant Consent?

Applying a codebook does not eliminate the ethical concerns associated with processing interview transcripts through AI.

Even when the model's task is limited to classification, the underlying data may contain personal experiences, sensitive disclosures, or indirect identifiers.

Researchers should verify whether the intended AI service and processing arrangements comply with ethics approval, participant consent, institutional policy, contractual restrictions, and applicable privacy law.

Where appropriate, institutionally approved or locally controlled processing environments may reduce some risks. Their suitability must still be assessed rather than assumed.

Testing Whether AI Applies a Codebook Correctly

Applying the Codebook to Interview Excerpts

The researcher asks AI to apply only the specified codes and permit multiple assignments when supported.

The assignments above represent the intended interpretation of the hypothetical codebook. They are not empirical evidence of any particular AI system's performance.

What if AI Misclassifies a Passage?

Suppose AI assigns CONNECTIVITY to the statement about sharing a laptop. The researcher should not simply correct that individual row and move on.

The error may indicate that the model is treating all difficulties accessing online classes as connectivity problems.

The researcher should inspect other passages involving device access, clarify the distinction if necessary, and test whether the system consistently follows the exclusion criteria.

Likewise, if the model assigns PARTICIPATION_ANXIETY whenever someone keeps a camera or microphone off, it may be inferring an emotional state that the participant did not report.

A Validation Workflow

A procedure that performs well on a small validation set may still encounter difficulties with different participants, language varieties, or interview contexts. Validation should therefore be treated as an ongoing quality-control activity rather than a one-time certification.

Common Mistakes in AI-Assisted Codebook Application

How to Apply a Qualitative Codebook With AI Responsibly

AI-assisted codebook application is most defensible when the task is clearly defined, the coding rules are sufficiently explicit, and the researcher can evaluate the resulting assignments.

The objective should not be to make the model classify every passage as quickly as possible. It should be to determine whether AI can apply the intended analytical distinctions with an acceptable level of error for the research purpose.

Use a Prompt That Enforces the Codebook Boundaries

A practical instruction might be:

"Apply only the codes defined in the supplied qualitative codebook. Follow each code's definition, inclusion criteria, exclusion criteria, and examples. For every transcript segment, return its identifier, the applicable code or codes, the exact supporting text, and a brief explanation linked to the coding rules. Use NO_CODE when no definition is satisfied and FLAG_FOR_REVIEW when the available context is insufficient. Do not invent codes, infer unstated participant experiences, or modify quotations."

This instruction makes the intended behavior explicit, but it does not guarantee that the model will comply. The resulting assignments still require validation.

Keep the Codebook and Coding Records Versioned

When definitions change, record what changed and why. A revised exclusion criterion may alter how previously coded passages should be classified.

Maintain stable identifiers for transcripts and coded segments. Preserve the codebook version, relevant model information, prompts, coding outputs, researcher corrections, and validation results.

These records make it possible to distinguish changes in the coding framework from changes in the model's behavior.

Separate Coding Consistency From Interpretation

Even when AI applies a codebook accurately, the resulting classifications do not automatically establish the study's findings.

Researchers must still examine relationships among coded passages, contextual differences, contradictions, and the substantive meaning of the material.

Depending on the methodology, this may contribute to developing themes from interview transcripts or another form of qualitative interpretation.

Document the AI contribution and follow applicable requirements for reporting AI use in qualitative data analysis. Responsibility for the final analytical claims remains with the researchers.

Before Using AI to Apply a Qualitative Codebook

Frequently Asked Questions About AI Qualitative Codebook Application

AI Can Apply a Codebook, but Coding Accuracy Must Be Demonstrated

How to Cite This Guide

This guide is intended to be read, shared, and used in research, teaching, and academic work. If you draw on its ideas, explanations, or other content, please acknowledge the source by citing the guide. Doing so gives appropriate credit and helps your readers locate the original resource.