03 · What You Need to Know
What Cognitive Interviewing and Pilot Testing Reveal That AI Cannot Establish
What Is Cognitive Interviewing in Questionnaire Development?
Cognitive interviewing is a qualitative pretesting method used to investigate how respondents understand and answer survey questions. Rather than simply asking whether an item is clear, the researcher examines the thought processes involved in producing an answer.
Willis (2005) describes cognitive interviewing as an approach to evaluating questionnaire design through investigation of respondents' interpretations and response processes.
For example, a researcher might ask a student to answer the item "I can independently complete challenging research tasks" and then explain what the word "independently" means to them.
One student might interpret it as completing a task without assistance. Another might interpret it as making decisions without direct supervision. These interpretations may have different implications for the construct being measured.
AI can predict possible interpretations, but only actual respondent feedback can establish which interpretations arise among the people participating in the pretest.
What Happens When Respondents Answer Survey Questions?
A widely used cognitive model of survey responding describes several processes involved in answering questions. Tourangeau, Rips, and Rasinski (2000) discuss comprehension, retrieval, judgment, and response selection as central components of survey response.
| Response Process |
What the Respondent Does |
Possible Questionnaire Problem |
| Comprehension |
Interprets the question and its terminology. |
Understands "AI use" differently from the researcher. |
| Retrieval |
Recalls relevant experiences or information. |
Cannot remember the number of times AI was used during the previous semester. |
| Judgment |
Evaluates the retrieved information to form an answer. |
Is uncertain how to combine experiences that differ across teaching activities. |
| Response selection |
Maps the answer onto the available response categories. |
Cannot distinguish between "Rarely" and "Sometimes." |
These processes help explain why a grammatically correct question can still produce problematic responses.
AI may identify plausible difficulties at each stage. However, its predictions do not demonstrate which problems actual respondents experience or how those problems affect their answers.
What Is the Difference Between Cognitive Interviewing and Pilot Testing?
The terms are sometimes used interchangeably, but they are not identical.
Cognitive interviewing
Investigates how respondents interpret questions and arrive at their answers, often using probing or think-aloud techniques.
Pilot testing
Rehearses the questionnaire or data collection procedures on a limited scale to identify problems before the main study.
A pilot study may examine questionnaire completion time, survey routing, missing responses, administration procedures, technical problems, and the feasibility of the proposed data collection process.
Cognitive interviewing focuses more directly on the response processes underlying individual answers.
Both can contribute to questionnaire development, although the specific activities and evidence collected differ.
What Can AI Simulated Respondents Actually Tell You?
Generative AI can imitate different respondent profiles and produce hypothetical answers. For example, a researcher could ask a model to respond as a first-year university student, an experienced instructor, or a participant unfamiliar with generative AI.
The resulting simulation may reveal obvious design problems. A hypothetical respondent might encounter a question that assumes prior AI use even though the simulated profile specifies no experience with AI.
This can help researchers identify missing response categories or inappropriate skip instructions.
However, the simulated response reflects the model's generated representation of a respondent, not an observed response from someone in the target population.
AI-generated respondents may also reproduce stereotypical assumptions about particular groups or generate answers that are more coherent and articulate than real survey responses.
Consequently, simulations are better understood as exploratory design exercises than as evidence of respondent comprehension or population behavior.
Why Generating Hundreds of Synthetic Responses Is Not a Pilot Study
A researcher might ask AI to generate 200 questionnaire responses and then calculate item statistics, reliability coefficients, or factor loadings from the resulting dataset.
Such calculations may be mathematically possible, but they describe the simulated data-generating process rather than the behavior of actual respondents.
If AI produces internally consistent answers, the resulting Cronbach's alpha may appear favorable. This does not establish that the questionnaire would produce similarly consistent responses among real participants.
Likewise, factor analysis of synthetic responses may produce an apparently coherent structure because the simulation was designed to reflect particular relationships.
Watch Out
Do not report AI-generated responses as though they were collected from actual pilot participants. Synthetic data may be useful for testing software, questionnaire logic, or analytical workflows, but they cannot independently establish the empirical measurement properties of a questionnaire in the intended population.
Researchers should distinguish simulated data from genuine pilot data in their records, analyses, and reporting.
How Can AI Help Prepare Cognitive Interviews?
AI can help researchers draft interview probes that explore how respondents interpret questionnaire items.
For example, an item about academic self-efficacy might be accompanied by probes such as:
- What does "challenging academic assignment" mean to you?
- What were you thinking about when you selected your answer?
- Was there any word or expression that you found difficult to understand?
- How did you decide between the two response categories you considered?
These probes can help researchers investigate possible comprehension and response-selection problems.
However, cognitive interviewers should not rely on AI-generated probes so rigidly that they overlook unexpected participant explanations. The purpose is to understand the respondent's interpretation, not to confirm the researcher's anticipated problems.
How Should You Prompt AI for Preliminary Questionnaire Pretesting?
A useful prompt should explicitly distinguish hypothetical simulations from genuine respondent evidence.
Illustrative AI Prompt
Preparing a Questionnaire for Cognitive Interviewing
"Review the following questionnaire intended for undergraduate students. Identify possible comprehension, recall, judgment, and response-selection difficulties. For each concern, explain why it may arise and suggest a neutral cognitive interview probe that could investigate the issue with actual respondents. Also identify potential problems with response options, instructions, and skip logic. Do not simulate findings or claim that respondents experience these problems. Treat your suggestions as hypotheses to examine during human pretesting."
This prompt positions AI as a tool for preparing an evidence-gathering procedure rather than a substitute for conducting it.
AI can also help organize de-identified cognitive interview notes, provided researchers verify the summaries against the original data and comply with relevant privacy requirements.
Can AI Help Test Questionnaire Logic Before a Pilot Study?
Yes. This is one area where AI-assisted simulation can be especially practical.
Suppose a questionnaire asks whether respondents have used generative AI for teaching. Those who answer "No" should skip questions about their satisfaction with AI tools.
AI can help construct hypothetical response paths to check whether the questionnaire directs respondents appropriately. It may identify cases where a nonuser is routed into an inapplicable question or where a required answer is missing.
For digital questionnaires, researchers can also use deterministic testing to verify that the programmed routing follows the specified rules.
These checks examine questionnaire implementation rather than the psychological or cognitive processes of actual respondents. They are useful preparation for pilot testing, not a replacement for it.
How Many Respondents Are Needed for Cognitive Interviewing or Pilot Testing?
There is no universal sample size that applies to every cognitive interview study or questionnaire pilot.
The appropriate number depends on the questionnaire's complexity, diversity of the target population, study purpose, and the problems the researcher seeks to identify.
Cognitive interviewing often uses purposive recruitment to obtain respondents who can reveal relevant differences in interpretation. The number of interviews may be adjusted as findings emerge and revisions are tested.
A pilot study designed to examine operational feasibility may have different sampling requirements from a preliminary psychometric investigation.
Researchers should justify their pretesting approach using relevant methodological guidance rather than assuming that a fixed number of simulated or actual respondents guarantees questionnaire quality.
What Should Researchers Do With Problems Identified During Pretesting?
Pretesting should inform decisions about whether to retain, revise, remove, or further investigate questionnaire items.
Suppose several respondents interpret a question about "independent research" as referring to working without classmates, while the researcher intends to measure autonomous research decision-making.
The researcher may revise the wording to clarify the intended task and conduct additional interviews to examine whether the revision resolves the problem.
AI can suggest alternative wording, but the decision should be grounded in the construct definition and actual respondent feedback.
This iterative process contributes to the broader work of evaluating questionnaire validity, particularly evidence concerning response processes.
