03 · What You Need to Know
How AI Categorization of Open-Ended Survey Responses Works
What Does It Mean to Categorize an Open-Ended Response?
An open-ended survey question allows respondents to answer in their own words rather than selecting from predetermined options. Categorization involves identifying what each response communicates and assigning it to one or more analytically meaningful groups.
Suppose students answer the question, "What makes online learning difficult for you?"
| Original Response |
Possible Category |
| "Our internet disconnects during lectures." |
Internet connectivity |
| "The course website is difficult to navigate." |
Platform usability |
| "My instructor takes several days to answer questions." |
Instructor communication |
| "I cannot concentrate because our house is noisy." |
Home learning environment |
These categories allow the researcher to examine recurring concerns across respondents. Depending on the study, the researcher may subsequently count how many participants mentioned each concern or investigate the experiences represented within a category.
The categories are not automatically themes in the methodological sense. A category such as "internet connectivity" describes a topic, whereas a qualitative theme may require a more developed interpretation of patterns and meanings across the dataset.
Should Researchers Create Categories Before or After Using AI?
There are two common approaches. The first is to develop categories before AI classification. The second is to examine the responses and develop categories through an exploratory process in which AI may offer suggestions.
| Approach |
How AI Is Used |
Principal Consideration |
| Predefined categorization |
AI assigns responses to researcher-supplied categories. |
Definitions must be sufficiently clear to distinguish categories. |
| Exploratory categorization |
AI suggests categories based on patterns it identifies in the responses. |
Suggested categories require researcher evaluation and refinement. |
| Hybrid categorization |
AI applies existing categories while identifying responses that may require additional ones. |
Changes to the framework must be documented and applied consistently. |
If your study already uses a theoretical framework or established survey classification scheme, predefined categories may be appropriate. For an exploratory study, you may need to examine participants' language before settling on the final categories.
The process of developing coding categories with AI deserves particular attention because a plausible category is not necessarily an analytically useful one.
Can One Survey Response Belong to Multiple Categories?
Yes. Many open-ended responses contain more than one idea, and forcing each answer into a single category can misrepresent the data.
Consider this response:
"The internet is unreliable, and I also find it difficult to ask questions because my instructor rarely responds."
The response addresses both internet connectivity and instructor communication. Assigning only the first category would conceal the second concern.
Single-label categorization
Each response receives one category, typically representing its primary issue.
Multi-label categorization
Each response may receive several categories when it contains multiple relevant ideas.
Neither approach is universally correct. Single-label categorization may be suitable when the research question specifically requires one primary classification and a defensible decision rule exists. Multi-label categorization is often preferable when the aim is to identify all concerns expressed by respondents.
The choice also affects interpretation. With multi-label coding, the percentages for individual categories may add up to more than 100% because a respondent can contribute to several categories.
Should AI Categorize Entire Responses or Individual Statements?
The unit of analysis matters. A short answer containing one clear idea may be classified as a whole. A longer response may contain several analytically distinct statements.
For example:
"Recorded lectures help me review difficult lessons. However, group activities are frustrating because some members do not participate, and I often have trouble submitting assignments through the platform."
This answer contains a positive experience with recorded lectures, a concern about group participation, and a platform submission problem.
A response-level classification could assign three labels to the same respondent. Alternatively, a segment-level approach could code each relevant passage separately.
Researchers should define whether they are counting respondents, responses, statements, or code occurrences. These quantities are not interchangeable, and confusing them can produce misleading frequency tables.
How Can Researchers Give AI Clear Categorization Instructions?
AI classification is easier to examine when the instructions specify what the model should classify, which labels are permitted, and how exceptions should be handled.
A practical instruction might read:
Illustrative AI Instruction
Classifying Student Survey Responses
"Classify each survey response using the supplied category definitions. Assign every category clearly supported by the respondent's words. Multiple categories are allowed. Do not infer experiences that are not expressed. Preserve the original response and its identifier. For each assignment, provide a short supporting quotation. If a response is ambiguous or does not fit the framework, mark it for human review. Return the results in a structured table."
This instruction establishes several useful constraints, but it does not guarantee compliance. Researchers must still inspect whether the model has omitted responses, altered quotations, or assigned unsupported categories.
When applying an established framework, a researcher-developed codebook should specify definitions, inclusion and exclusion criteria, and how overlapping cases are handled.
How Should AI Handle Answers That Do Not Fit Any Category?
A classification system should not require every answer to fit an existing category when the evidence does not support that decision.
Suppose the available categories concern internet connectivity, platform usability, and instructor communication, but a respondent writes:
"My biggest difficulty is finding a quiet place to study after my evening shift."
The answer may concern employment schedules, the study environment, or both. Forcing it into internet connectivity would create an inaccurate classification.
Researchers can provide options such as "other," "unclear," or "requires review," but these labels should have distinct meanings. "Other" may represent a clearly understood concern outside the framework, while "unclear" indicates insufficient information to classify confidently.
AI can assist in identifying ambiguous coding cases, although researchers must evaluate whether the model recognizes the relevant uncertainty.
Can Researchers Use AI Categories to Calculate Frequencies and Percentages?
Yes, once the classification procedure has been adequately evaluated and the unit of counting is clear. Categorized responses can support descriptive summaries, although numerical outputs inherit any errors in the underlying assignments.
For example, if 80 out of 200 respondents mention internet connectivity, the researcher may report that 40% of respondents mentioned connectivity problems.
Researchers should distinguish category prevalence from the frequency of individual statements. One respondent might mention the same concern several times, but that does not necessarily justify counting the respondent repeatedly.
How Accurate Is AI Categorization?
There is no single accuracy rate that applies across all open-ended surveys. Performance depends on the model, prompt, language, category definitions, response length, and difficulty of the classification task.
Gilardi, Alizadeh, and Kubli (2023) examined ChatGPT's performance on several text-annotation tasks and reported encouraging results relative to human annotators in their evaluated settings. Their findings support the potential of large language models for structured annotation, but do not establish that every survey coding task will perform equally well.
Parfenova and colleagues (2025) compared language models and human experts in inductive coding and found that task difficulty influenced model performance. This is particularly relevant when responses contain complex meanings that cannot be resolved through simple topic identification.
Researchers should evaluate performance on their own material rather than adopting accuracy claims from unrelated studies. They should also examine whether errors disproportionately affect particular respondent groups, response styles, or categories.
Importantly, consistent classification does not necessarily mean valid classification. A model may repeatedly assign the same incorrect category.
How Much Human Verification Is Necessary?
Human verification should be planned according to the complexity of the task and the consequences of errors. There is no universally defensible rule requiring researchers to inspect a fixed percentage of responses.
For a structured classification task, researchers might independently review a representative evaluation sample, include less frequent categories, examine ambiguous cases, and investigate disagreement patterns. If the model performs poorly, more extensive review or a revised coding approach may be necessary.
The extent of manual verification should therefore be justified by evidence about coding performance rather than convenience alone.
What Data Protection Issues Should Researchers Consider?
Open-ended survey responses may contain names, locations, health information, workplace details, or other potentially identifying material that respondents voluntarily disclose.
Before submitting these responses to an AI system, researchers should verify whether the proposed processing is permitted by participant consent, ethics approval, institutional policies, applicable law, and the service's data handling arrangements.
Watch Out
Removing names does not necessarily make open-ended survey responses anonymous. Respondents may describe unique circumstances that permit re-identification. Use an approved processing environment and avoid uploading restricted material when appropriate safeguards or authorization are unavailable.
04 · A Practical Example
Categorizing 500 Student Responses About Online Learning
Hypothetical Example
From Written Answers to an Interpretable Category Summary
A researcher collects 500 usable responses to the question, "What challenges do you experience when participating in online classes?" The objective is to describe the concerns students report, rather than establish causal relationships or produce an interpretive thematic analysis.
1. Examine the responses.
The researcher reads a varied initial selection and identifies provisional categories reflecting the survey question.
2. Define the categories.
Each category receives inclusion criteria, exclusion criteria, and examples. Multiple codes are permitted.
3. Pilot AI classification.
The model categorizes a pilot set while preserving response identifiers and providing supporting text.
4. Evaluate the assignments.
The researcher compares the outputs with independently reviewed material and examines problematic classifications.
5. Finalize the procedure.
After justified revisions and further evaluation, the researcher applies the documented procedure to the remaining responses and conducts the required verification.
6. Summarize the results.
The researcher counts distinct respondents within each category and interprets the results alongside the original answers.
What Might the Resulting Summary Look Like?
Suppose the verified classifications produce the following hypothetical distribution:
| Category |
Respondents |
Percentage |
| Internet connectivity |
210 |
42% |
| Time and scheduling constraints |
165 |
33% |
| Platform usability |
120 |
24% |
| Home learning environment |
105 |
21% |
| Instructor communication |
80 |
16% |
The percentages exceed 100% when combined because respondents could mention several challenges. The counts describe how many respondents mentioned each category, not how many considered it their single greatest difficulty.
The researcher might conclude that connectivity was the most frequently mentioned challenge in this sample. However, the results alone would not establish that connectivity causes poorer academic performance or that it is the most consequential problem.
The categories become useful research evidence only when the coding procedure is defensible and the interpretation respects what the data can actually support.
