Can Generative AI Apply a Human-Developed Research Codebook?

Can AI Follow the Coding Rules You Have Already Developed?

You have spent considerable time developing a research codebook. Each code has a definition, inclusion criteria, exclusion criteria, and examples. Your next task is to apply those rules to hundreds of interview excerpts or open-ended survey responses.

Instead of asking generative AI to invent categories, could you give it your codebook and instruct it to classify the data accordingly?

The distinction is consequential. A researcher-developed codebook establishes what the categories are supposed to mean. Yet even with explicit instructions, an AI system may misunderstand boundaries, assign unsupported labels, or apply the same definition differently across responses.

Yes, AI Can Apply a Codebook, but Following Instructions Must Be Verified

How Generative AI Applies a Researcher-Developed Codebook

What Is a Research Codebook?

A research codebook is a documented framework describing the codes used to classify or interpret data. Depending on the methodology, it may contain code names, operational definitions, inclusion and exclusion criteria, illustrative excerpts, and instructions for handling overlapping or uncertain cases.

For example, a researcher investigating teachers' experiences with educational technology might develop a codebook containing categories such as technical infrastructure, institutional support, instructional efficiency, and pedagogical concerns.

The purpose of the codebook is not simply to list topics. It establishes the distinctions researchers intend to make when examining the data.

A codebook is especially important when coding decisions must be applied consistently across numerous observations or evaluated against predetermined criteria. Not every qualitative methodology requires a fixed codebook, however. Some interpretive approaches use evolving codes and researcher reflexivity rather than standardized classification rules.

How Is Applying a Codebook Different From Generating Codes?

When AI generates codes, it proposes labels based on its interpretation of the material. When AI applies a codebook, the researcher supplies the permissible categories and their meanings.

The distinction helps researchers avoid treating code discovery and code application as the same task. The process of generating coding categories automatically requires different forms of evaluation from applying a framework whose definitions have already been established.

What Information Should a Codebook Contain Before AI Uses It?

A code name alone is often insufficient. The model needs enough information to distinguish the intended meaning of one category from neighboring categories.

Consider a code named "technical barriers." Without a definition, AI might apply it to unreliable internet connections, insufficient computer skills, confusing interfaces, or inadequate technical support.

These issues may be related, but a particular study might need to distinguish them.

Negative examples are useful because they demonstrate what a code does not mean. They can help clarify boundaries that might otherwise be inferred too broadly.

Researchers should also specify whether multiple codes may apply, what constitutes a coding unit, and how to handle responses that do not match any definition.

Can AI Follow Inclusion and Exclusion Criteria?

Generative AI can be instructed to follow explicit inclusion and exclusion criteria, but its compliance must be tested.

Suppose the codebook distinguishes "lack of training" from "lack of technical support."

A statement such as "Nobody taught us how to use the platform" would ordinarily support lack of training. By contrast, "The platform stopped working, and nobody was available to fix it" would support lack of technical support.

The distinction becomes more difficult when a participant says, "We were left to figure everything out ourselves." The statement may indicate insufficient guidance, but it does not necessarily establish whether the problem concerns training, troubleshooting, or another form of support.

A well-designed procedure should allow AI to flag insufficient evidence instead of forcing a code. Researchers can then inspect the surrounding interview context or classify the passage as unresolved.

Should the AI Be Allowed to Create New Codes?

That depends on the research design. If the task is strictly deductive classification, the model should generally use only the approved codebook labels. Otherwise, different responses could be classified using an expanding and undocumented set of categories.

However, researchers may also want to identify meaningful material that the existing framework does not capture. A useful compromise is to instruct AI to apply approved codes while separately flagging passages that appear to require consideration of a new category.

The model should not silently add those categories to the official coding framework. Any revision should be evaluated, documented, and applied appropriately to previously processed material.

Can AI Apply Multiple Codes to the Same Passage?

Yes. A response may legitimately satisfy several code definitions.

For example, "I cannot attend online classes because my internet is unreliable and my work schedule conflicts with lectures" may support both internet connectivity barriers and scheduling constraints.

The codebook should explain whether codes are mutually exclusive, whether multiple assignments are allowed, and whether a primary code is required.

This is especially important when categorizing open-ended survey responses, where respondents frequently mention several concerns in a single answer.

How Should Researchers Write Instructions for Codebook-Based AI Coding?

A useful prompt should specify the analytical task, provide the complete relevant codebook, define the output format, and establish what the model should do when the evidence is insufficient.

The researcher should test this instruction against difficult examples. A prompt that appears precise may still produce incorrect assignments when categories overlap or respondents use indirect language.

How Can Researchers Evaluate Whether AI Followed the Codebook?

For structured coding tasks, researchers can compare AI assignments with independently reviewed reference data. The reference set should represent the intended coding task and include difficult cases rather than only obvious examples.

Several evaluation measures may be useful:

• Precision: Among assignments made to a category, how many are supported by the reference coding?
• Recall: Among passages that should receive a category, how many did AI identify?
• F1 score: A combined measure balancing precision and recall.
• Agreement measures: Statistics such as Cohen's kappa may be appropriate for particular coding designs, provided their assumptions are satisfied.

Overall accuracy alone can be misleading when categories are unevenly distributed. A model may classify a common category well while repeatedly missing a less frequent but analytically important one.

Researchers should inspect category-specific errors and determine whether disagreement results from model behavior, unclear definitions, or limitations in the reference coding.

Gilardi, Alizadeh, and Kubli (2023) demonstrated the potential of ChatGPT for text annotation in evaluated classification tasks. Their findings are encouraging, but performance established in one annotation setting should not be assumed to transfer automatically to a different research codebook.

Does High Agreement Mean the Coding Is Valid?

Not necessarily. A model may reproduce a flawed codebook consistently, or both AI and human coders may apply an inappropriate category definition.

Agreement evaluates aspects of classification consistency relative to a reference. Validity also concerns whether the codes represent the concepts the study intends to examine.

Researchers should therefore distinguish consistent coding from valid coding. Strong agreement is useful evidence in some methodological approaches, but it does not independently establish conceptual adequacy.

What Does the Research Literature Suggest?

Published research indicates that language models can perform structured annotation tasks, although performance depends on the task, instructions, and evaluation design.

Gilardi and colleagues (2023) compared ChatGPT with crowd workers across several text-annotation tasks and found strong performance in their evaluated settings. This demonstrates the potential of language models for classification, not universal superiority over trained research coders.

Parfenova and colleagues (2025) examined inductive text coding with language models and human experts, highlighting difficulties associated with more complex coding material. Although inductive category development differs from applying a fixed codebook, the findings reinforce the need to evaluate models against the demands of the specific task.

The broader methodological literature also cautions against treating all qualitative coding as standardized classification. Braun and Clarke (2021) explain that different approaches to thematic analysis involve different assumptions about coding and quality. A codebook-based AI workflow is therefore more directly aligned with some analytical traditions than others.

How Should Researchers Document AI Codebook Application?

Documentation should identify the codebook version, model or service used, relevant settings, prompt instructions, preprocessing decisions, coding units, evaluation procedure, and any subsequent revisions.

Researchers should retain the original text and final approved assignments. If the codebook changes, the records should show when and why the revision occurred.

These details support reproducible AI-assisted coding, although exact model outputs may still vary across runs or service versions.

Before submitting data to an external AI system, researchers must also verify that the processing arrangement complies with consent, ethics approval, institutional requirements, applicable law, and relevant confidentiality obligations.

Applying a Research Codebook to Students' Online Learning Responses

Step 1: Supply the Approved Codebook

The researcher provides definitions and decision rules for each category, including examples that distinguish technical connectivity problems from platform navigation difficulties.

The AI is instructed to use only the four approved codes, permit multiple assignments, and flag passages that cannot be classified adequately.

Step 2: Pilot the Coding Procedure

The model processes an initial set of responses. Its proposed assignments are compared with independently reviewed coding decisions.

Step 3: Investigate the Errors

The fourth response reveals that the model introduced a category not permitted by the codebook. The fifth shows that the model overlooked a second applicable category.

The researcher revises the instructions to require exact code identifiers and explicit consideration of every applicable category. The revised procedure is tested again using responses not employed to develop the revision.

The researcher does not simply assume that a more detailed prompt has corrected the problem. The new outputs must demonstrate that the intended rules are being followed.

Step 4: Apply the Evaluated Procedure

Once the pilot demonstrates performance acceptable for the study's purpose, the researcher processes the remaining responses using the documented workflow.

Responses flagged as ambiguous, assignments involving consequential categories, and an appropriately selected evaluation sample receive further review. If substantial errors remain, the researcher may need to revise the framework or return to more extensive human coding.

The final dataset preserves response identifiers, original text, assigned codes, review status, and researcher-approved decisions.

The codebook establishes the intended classification rules. The evaluation establishes whether AI has applied them adequately. Both are necessary for a defensible structured coding procedure.

Common Mistakes When Asking AI to Apply a Codebook

When Is AI Codebook Application Worth Using?

AI-assisted codebook application may be useful when researchers have explicit classification criteria and a sufficiently large dataset to justify developing and validating an automated workflow.

Its value is less obvious when the dataset is small, the categories remain conceptually unstable, or the methodology requires ongoing interpretive engagement rather than standardized classification.

Researchers should also establish how much manual checking of AI-coded material is necessary for their particular task. A fixed percentage cannot substitute for evaluating the kinds of errors the model makes.

The practical advantage of a codebook-based workflow is that its decisions can be examined against explicit criteria. The limitation is that explicit criteria do not eliminate ambiguity, interpretive judgment, or the possibility of systematic error.

Before Asking AI to Apply Your Research Codebook

Frequently Asked Questions About AI Codebook Application

A Codebook Gives AI Rules, but Evaluation Establishes Whether It Followed Them

How to Cite This Guide

This guide is intended to be read, shared, and used in research, teaching, and academic work. If you draw on its ideas, explanations, or other content, please acknowledge the source by citing the guide. Doing so gives appropriate credit and helps your readers locate the original resource.