How Much AI-Coded Research Data Should Be Manually Checked?

Should You Manually Check 10%, 20%, or Every AI-Coded Response?

You have used generative AI to code 5,000 open-ended survey responses. The model followed your codebook, produced a complete classification table, and flagged some responses as uncertain. You now need to determine how much of the output should be checked against the original data.

Would reviewing 10% be enough? Should you inspect 20%, or perhaps manually verify every response? Researchers sometimes choose a convenient percentage without considering what that sample can actually establish.

The difficulty is that verification serves several purposes. You may want to estimate the overall error rate, identify categories that AI frequently misclassifies, or detect errors that could materially change your findings. These objectives require different approaches to selecting and examining responses.

There Is No Universal Percentage for Manual Verification

How to Determine an Appropriate Manual Verification Strategy

Why Is Checking a Fixed Percentage Not Enough?

A percentage describes how much material is reviewed, but it does not explain how the material was selected or what conclusions can be drawn from the review.

For example, checking 10% of 100 responses means reviewing only ten cases. Checking 10% of 20,000 responses means reviewing 2,000 cases. The same percentage produces very different amounts of evidence.

Even when the number reviewed is substantial, the sample may be misleading if it contains mostly straightforward responses or excludes less frequent categories.

A defensible verification strategy begins by specifying the question the review is intended to answer. Estimating overall accuracy, detecting systematic errors, and resolving ambiguous cases are related objectives, but they are not identical.

What Are Researchers Actually Trying to Verify?

Manual verification can assess several properties of AI-coded data.

These checks provide different kinds of evidence. A review designed to estimate overall accuracy may contain too few rare-category cases to evaluate them adequately. Conversely, a sample deliberately enriched with difficult cases cannot be treated as representative of the entire dataset without accounting for its selection design.

Should You Use Random Sampling or Targeted Review?

Both can be useful, but they answer different questions.

A simple random sample gives each eligible response a known and equal probability of selection. When properly implemented, it can support estimates of overall coding performance, subject to sampling uncertainty and the quality of reference coding.

Targeted review deliberately selects responses that are more likely to be problematic, such as ambiguous statements, rare categories, code-switched responses, or assignments associated with known model weaknesses.

A combined strategy is often useful. Researchers may independently review a probability sample to estimate performance, then conduct additional targeted checks to investigate consequential errors.

Results from the two components should be reported separately unless the sampling design and analysis justify combining them.

How Do You Calculate a Verification Sample Size?

For a structured classification task, researchers may estimate the proportion of correct assignments using a probability sample. A conventional starting point for planning the sample size is the formula for estimating a population proportion.

When sampling without replacement from a finite dataset, a finite population correction may reduce the required sample size.

These formulas assume an appropriate probability-sampling design and a binary outcome such as agreement or disagreement with a reference classification. They are planning tools, not universal requirements for AI coding verification.

Actual analysis should use confidence intervals appropriate to the sampling design and observed data. If responses are clustered within institutions, interviewees, or other units, the independence assumptions underlying simple formulas may not hold.

Does a 95% Confidence Level Mean You Can Trust 95% of the AI Codes?

No. A 95% confidence level describes the long-run coverage properties of a confidence interval procedure under its assumptions. It does not mean that 95% of the AI classifications are correct.

Similarly, a margin of error of five percentage points concerns uncertainty in an estimated population proportion. It does not mean the model is allowed to make only five percent errors.

Researchers should distinguish statistical confidence in an estimated performance measure from confidence in the correctness of individual coding decisions.

This distinction is especially relevant when interpreting AI-generated confidence scores, which may not represent calibrated probabilities of correctness.

Why Might a Representative Sample Still Miss Important Errors?

Suppose only 2% of responses concern accessibility barriers. A random sample of 100 responses would contain approximately two such responses on average, assuming the sample reflects the overall distribution.

That may be insufficient to estimate how well the model identifies accessibility-related concerns.

Researchers may therefore need additional sampling designed to capture rare or consequential categories. This could involve reviewing responses independently identified as potentially relevant, enriching the sample with cases from particular subgroups, or examining known difficult response types.

Care is needed when the categories used for stratification come from the AI itself. Sampling only records that AI labeled as accessibility-related cannot reveal all accessibility cases the model failed to identify.

Category-specific evaluation should therefore consider both false positives and false negatives, using a reference procedure capable of identifying missed cases.

Should AI-Flagged Ambiguous Cases Always Be Reviewed?

Flagged cases often warrant attention because they may involve insufficient information, overlapping categories, or uncertainty about the appropriate interpretation.

However, reviewing only flagged cases is inadequate unless the researcher has established that the flagging procedure captures the errors of concern.

A model may confidently assign an incorrect code without recognizing the problem. Independent review of some unflagged responses helps estimate how frequently such errors occur.

The role of AI ambiguity detection is therefore to support review prioritization, not to replace evaluation of the unflagged material.

How Should Researchers Establish the Human Reference Coding?

The quality of manual verification depends on the quality of the human reference decisions.

For structured coding, researchers should establish clear definitions, train reviewers where appropriate, and document how disagreements are resolved. Independent coding before exposure to AI suggestions can reduce the risk that the model's labels influence reference judgments.

Human coders may also make mistakes. Where coding decisions remain disputed, researchers should examine the original text, relevant context, and category definitions rather than automatically treating one person's classification as correct.

In interpretive qualitative approaches, a fixed reference classification may not be methodologically appropriate. Verification may instead focus on whether AI suggestions preserve contextual meaning and contribute to a coherent, reflexive analytical process.

Braun and Clarke's methodological work emphasizes that qualitative approaches differ in their assumptions about coding and quality. A validation procedure suitable for deductive content classification should not automatically be imposed on reflexive thematic analysis.

Can You Stop Reviewing Once the AI Reaches a Particular Accuracy?

There is no universal accuracy threshold at which human review becomes unnecessary.

Suppose an AI system achieves 95% overall accuracy. Whether that is acceptable depends on the intended analysis and the distribution of errors.

If the remaining mistakes are concentrated in a category central to the study's conclusions, the overall percentage may be misleading. A model that consistently overlooks accessibility concerns or misattributes institutional problems to individuals could distort important findings despite high aggregate performance.

Researchers should define performance requirements in relation to the study's objectives and evaluate category-specific errors before deciding whether the remaining risk is acceptable.

The distinction between coding consistency and coding validity remains important. High agreement with a reference set does not independently establish that the codebook adequately represents the intended concepts.

Should Verification Continue After the Initial Evaluation?

Sometimes. Initial validation establishes performance under the conditions represented in the evaluation data. Subsequent changes in the model, codebook, prompt, respondent population, or language characteristics may affect that performance.

For large-scale or continuing coding operations, researchers may conduct periodic checks or evaluate new batches when the processing conditions change.

Documentation should identify which records were reviewed, how they were selected, what errors were found, and what corrective actions followed.

When coding rules are revised, affected records may need to be reconsidered to ensure that different parts of the dataset are not classified under incompatible definitions.

What Does the Research Literature Support?

Gilardi, Alizadeh, and Kubli (2023) demonstrated that language models can perform text-annotation tasks effectively in certain evaluated settings. Their results support the feasibility of AI-assisted classification but do not establish a universal percentage of outputs requiring human verification.

Artstein and Poesio (2008) provide a detailed methodological discussion of inter-coder agreement and its interpretation in computational linguistics. Their work is relevant when researchers compare AI and human coding decisions.

Cochran's Sampling Techniques provides established principles for probability sampling and sample-size estimation. These principles can support a statistically defensible evaluation design when the objective is to estimate classification performance across a defined dataset.

The combined implication is that verification should be designed around a clearly specified inferential or analytical objective. A convenient inspection percentage, by itself, provides little assurance of coding quality.

Designing a Manual Verification Plan for 5,000 AI-Coded Responses

Step 1: Select a Probability-Based Evaluation Sample

Using the conservative sample-size assumptions described earlier, the researcher calculates an adjusted sample size of approximately 358 responses for the finite dataset of 5,000.

The 358 responses are selected using a documented simple random sampling procedure. Human reviewers classify them independently of the AI suggestions before the results are compared.

Step 2: Examine the Overall Classification Results

Suppose the review produces the following hypothetical results:

The observed agreement is approximately 91.9%. Under the assumed simple random sampling design, the researcher can calculate an appropriate confidence interval for the estimated population agreement rate.

The result does not establish that every unreviewed response is correctly classified, nor does it guarantee acceptable performance for each category.

Step 3: Investigate Rare and Consequential Categories

Only a small number of accessibility-related cases appear in the random sample. The researcher determines that the sample is insufficient to evaluate that category adequately.

An additional targeted review is therefore conducted using a strategy designed to identify possible accessibility cases, including responses not labeled as accessibility-related by AI.

The targeted findings are reported separately from the probability-sample estimate unless the sampling design supports a valid combined analysis.

Step 4: Examine the Consequences of Errors

Suppose the additional review reveals that AI sometimes classifies accessibility-related complaints as general platform usability problems.

The researcher investigates whether the error reflects unclear category definitions or model limitations. If the procedure is revised, the updated version is evaluated on additional data before affected assignments are finalized.

Step 5: Justify the Final Verification Decision

The researcher documents the probability-sampling procedure, reference coding, observed agreement, category-specific findings, and any additional review.

Whether the remaining assignments can be used without complete manual recoding depends on the study's purpose and the acceptability of the observed errors.

The verification plan is defensible because the sample was selected to answer specified questions about coding performance, not because the researcher followed an arbitrary percentage.

Common Mistakes When Verifying AI-Coded Research Data

How to Choose a Verification Strategy for Your Study

The most useful starting point is to identify what would make the AI-coded results unacceptable for your intended analysis. From there, choose a review procedure capable of detecting or estimating those problems.

Researchers should distinguish verification from the broader question of using AI coding without individual human review. A study may not require manual recoding of every response, but that decision should follow evidence about the coding procedure rather than precede it.

The verification plan should also consider whether systematic AI coding bias could distort particular categories or respondent groups. A sample designed only to estimate overall agreement may not be sufficient to detect such patterns.

Before Deciding How Much AI-Coded Data to Check

Frequently Asked Questions About Manual Verification of AI Coding

Verify Enough Data to Answer the Right Quality Question

How to Cite This Guide

This guide is intended to be read, shared, and used in research, teaching, and academic work. If you draw on its ideas, explanations, or other content, please acknowledge the source by citing the guide. Doing so gives appropriate credit and helps your readers locate the original resource.