03 · What You Need to Know
How Cognitive Interviewing Works in Questionnaire Development
What Exactly Is Cognitive Interviewing?
Cognitive interviewing is a research method used to evaluate survey questions by examining the meanings and thought processes respondents use when answering them.
Rather than simply recording a participant's selected answer, the researcher investigates how that answer was produced.
For example, after a respondent selects "Sometimes" for a question about verifying online information, the interviewer may ask what the respondent considers verification and which experiences they recalled.
The resulting explanation may reveal whether the respondent understood the intended behavior, used an unexpected reference period, or selected an answer that did not adequately represent their experience.
Cognitive interviewing has an established methodological tradition in survey research. Willis describes its application to questionnaire design, while the United States National Center for Health Statistics uses cognitive interviewing as a major approach to evaluating survey questions.
The method is particularly valuable because a questionnaire can appear clear to researchers while producing inconsistent interpretations among respondents.
What Cognitive Processes Are Involved in Answering a Survey Question?
A widely used framework in survey methodology distinguishes comprehension, retrieval, judgment, and response formulation.
These processes provide a useful structure for examining where questionnaire problems arise. They are analytical categories rather than a claim that every respondent consciously follows four separate stages in a fixed sequence.
| Response Process |
What the Respondent Does |
Possible Measurement Problem |
| Comprehension |
Interprets the question and its terminology |
Understands "academic engagement" differently from the intended definition |
| Retrieval |
Recalls information or experiences needed to answer |
Cannot remember activities over the specified recall period |
| Judgment |
Evaluates recalled information and constructs an answer |
Uses an unintended comparison standard |
| Response formulation |
Maps the judgment onto an available response option |
Finds that no response category represents the intended answer |
A single item can create difficulties at several stages. For example, a respondent may misunderstand the activity, struggle to remember its frequency, and then find the available categories insufficiently precise.
What Is the Think-Aloud Technique?
Think-aloud interviewing asks respondents to verbalize their thoughts while answering a question or completing a questionnaire.
A researcher might instruct participants to say what comes to mind as they read each question and decide on an answer, without attempting to provide a polished explanation.
This technique can reveal hesitation, uncertainty, alternative interpretations, and difficulties with response selection.
However, thinking aloud may feel unnatural to some participants. Others may provide only brief comments or become distracted by the additional task of verbalizing their thoughts.
The technique may also influence the response process it is intended to investigate.
Researchers should therefore provide suitable practice and avoid assuming that silence indicates effortless comprehension.
What Is Verbal Probing?
Verbal probing involves asking follow-up questions to investigate how a respondent understood or answered an item.
Probes may be planned in advance or developed during the interview in response to an observed difficulty.
For example, after a respondent answers a question about academic self-efficacy, the interviewer might ask what the term "challenging academic task" means to them.
Probes should investigate interpretation without supplying the researcher's preferred answer.
A leading probe such as "You understood this as your ability to complete difficult assignments, correct?" may encourage agreement rather than reveal the respondent's original interpretation.
Neutral probes generally provide more useful evidence about spontaneous meaning-making.
How Do Think-Aloud and Verbal Probing Differ?
Think-aloud interviewing
Respondents verbalize their thinking while completing the questionnaire, allowing researchers to observe aspects of the ongoing response process.
Verbal probing
Interviewers ask targeted questions about respondents' interpretations, recall, judgments, or response choices, either during or after item completion.
The approaches can be combined. A researcher may begin with think-aloud responses and use neutral probes when a participant hesitates or reveals an unexpected interpretation.
Neither method provides complete access to cognition. Respondents may be unable to articulate some processes, and retrospective explanations may be reconstructed after the answer has already been given.
What Questions Should a Cognitive Interviewer Ask?
Useful probes focus on the specific cognitive process being investigated.
| Purpose |
Illustrative Neutral Probe |
| Comprehension |
"What does the phrase 'evaluate information critically' mean to you?" |
| Paraphrasing |
"Could you explain this question in your own words?" |
| Retrieval |
"What experiences were you thinking about when you answered?" |
| Reference period |
"What period of time did you consider?" |
| Judgment |
"How did you decide whether that happened frequently?" |
| Response selection |
"How did you choose between 'Sometimes' and 'Often'?" |
| Applicability |
"Was there anything about this question that did not fit your experience?" |
Probes should be adapted to the question and respondent. Asking every possible probe after every item can make the interview unnecessarily burdensome and may alter how respondents approach later questions.
When Should Cognitive Interviewing Be Conducted?
Cognitive interviewing is especially useful after an initial item pool has been developed and conceptually reviewed, but before the instrument is finalized for substantive data collection.
It may also be appropriate during translation, cultural adaptation, mode changes, or revisions to response categories and recall periods.
For emerging constructs, interviews with the target population may contribute to earlier conceptual development, although exploratory construct interviews and cognitive interviews serve different purposes.
The approach described in Artino and colleagues' questionnaire-development framework places cognitive interviewing after item development and expert feedback and before pilot testing. This is a defensible sequence, but not the only possible workflow.
Researchers should select the timing according to which measurement decisions remain unresolved.
Is Cognitive Interviewing Necessary for Every Questionnaire?
Not necessarily. The need depends on the available response-process evidence and the consequences of misunderstanding.
A newly developed questionnaire addressing an unfamiliar or culturally sensitive construct may benefit substantially from cognitive interviewing.
An established instrument administered without modification to a closely comparable population may already have relevant evidence, although researchers should examine whether that evidence applies to their intended use.
Changes in language, population, administration mode, or item wording may create new response-process questions.
The decision should therefore be risk-based rather than governed by a universal requirement.
Who Should Participate in Cognitive Interviews?
Participants should resemble the intended respondents in characteristics relevant to how they understand and answer the questionnaire.
These may include language proficiency, educational background, professional experience, age, familiarity with the topic, and accessibility needs.
Purposeful selection is often appropriate because the aim is to identify potential response problems, not estimate their population prevalence.
For example, a questionnaire intended for students in asynchronous courses might include participants with different levels of online learning experience and familiarity with institutional terminology.
The research team should avoid relying exclusively on highly educated colleagues or research assistants who already understand the instrument's theoretical vocabulary.
This connects directly to involving target-population members in instrument review.
How Many Cognitive Interviews Are Needed?
There is no universally sufficient number.
The CDC's National Center for Health Statistics describes cognitive interview studies as typically involving relatively small purposive samples, often around 20 to 50 respondents in its methodological practice. This is descriptive guidance, not a universal minimum or requirement for every project.
Some development studies use smaller iterative rounds, especially when testing a limited set of questions. Others need broader recruitment because the instrument is complex or the population is heterogeneous.
Sample adequacy depends on the diversity of relevant respondent experiences, the number of questions, the problems being investigated, and whether additional interviews continue to reveal important difficulties.
A fixed sample size cannot guarantee that all consequential interpretation problems have been identified.
How Do You Conduct a Cognitive Interview?
Begin with a protocol specifying the questionnaire items, research objectives, planned probes, recording procedures, and method of analyzing responses.
Explain to participants that the purpose is to evaluate the questions rather than their knowledge or personal competence.
Ask participants to answer items in a manner that approximates the intended administration, while using think-aloud or probing procedures to investigate their responses.
Record the selected answer alongside relevant explanations, hesitation, clarification requests, and observed difficulties.
When the interview reveals a problem, avoid immediately teaching the participant the intended meaning and then treating their corrected answer as evidence that the original item worked.
Appropriate consent, privacy, and recording procedures should be followed according to the study context.
Should Cognitive Interviews Be Conducted Online or Face to Face?
Both may be appropriate.
Face-to-face interviews can support observation of questionnaire interaction and may facilitate detailed probing. Remote interviews can improve accessibility and allow participants to complete a digital questionnaire in a familiar environment.
However, remote sessions may introduce technical difficulties or limit observation of how participants interact with the instrument.
The choice should reflect the questionnaire's intended administration mode, participant accessibility, and the evidence needed.
When evaluating a mobile survey, observing participants using a mobile interface may reveal problems that would not appear when testing a printed copy.
How Should Cognitive Interview Findings Be Analyzed?
Cognitive interviewing produces qualitative evidence that requires systematic analysis rather than informal impressions.
Researchers can organize findings by questionnaire item and response-process category, comparing how different respondents interpreted the same wording.
The CDC describes an analytical approach that moves from individual interview evidence to comparisons across respondents and groups, followed by conclusions about question performance.
A practical analysis matrix might contain the item identifier, intended meaning, observed interpretation, type of difficulty, affected respondents, supporting evidence, and proposed revision.
Researchers should distinguish a clearly demonstrated misunderstanding from a possible problem inferred from hesitation alone.
Contradictory interpretations are valuable evidence, especially when they occur across population subgroups.
How Do You Decide Whether an Item Needs Revision?
Not every unusual response warrants a change.
Researchers should examine whether the difficulty could alter the intended measurement, whether it is attributable to the wording or administration, and whether it appears relevant to the target population.
For example, an unfamiliar technical term that repeatedly produces unintended interpretations may require revision.
By contrast, a respondent's genuine disagreement with a statement is not a questionnaire defect.
Revisions should preserve the construct definition and be reviewed for unintended consequences.
Where changes are substantive, another cognitive interview round may be needed to establish whether the revised item functions more appropriately.
Can Cognitive Interviewing Detect Problems With Response Options?
Yes. Respondents may understand a question but struggle to map their intended answer onto the available categories.
For example, a frequency question may offer "Never," "Rarely," "Sometimes," "Often," and "Always," while respondents use different personal standards for these labels.
Cognitive interviewing can reveal whether categories are distinguishable, whether a midpoint is interpreted as intended, and whether respondents need a legitimate not-applicable response.
However, participant preference alone does not determine the optimal scale format.
The separate question of how many response options a questionnaire item should have also requires consideration of the construct, measurement precision, and empirical evidence.
How Is Cognitive Interviewing Different From Pilot Testing?
Cognitive interviewing investigates the processes underlying individual responses.
Pilot testing may examine the complete administration workflow, completion rates, survey timing, missing responses, scoring, and preliminary measurement characteristics.
A pilot can reveal that respondents frequently skip an item without establishing why. Cognitive interviews may show that the question assumes an experience unavailable to some participants.
Both methods can be integrated into one development program, but neither automatically replaces the other.
Similarly, expert review of questionnaire items contributes conceptual judgments that cognitive interviews do not independently establish.
Can Cognitive Interviewing Establish Questionnaire Validity?
Cognitive interviewing can contribute important validity evidence based on response processes.
It may demonstrate that respondents interpret key terms consistently with the intended construct or reveal problems that undermine such interpretations.
However, it does not independently establish score reliability, internal structure, relationships with other variables, or equivalence across groups.
Furthermore, the absence of observed problems in a limited set of interviews does not prove that all future respondents will understand the questions correctly.
The findings should be integrated with theoretical, content-related, and quantitative evidence appropriate to the instrument's intended use.
How Should Cognitive Interviewing Be Reported?
Reports should describe why cognitive interviewing was conducted, how participants were selected, which techniques were used, and how the resulting evidence was analyzed.
Include the number and characteristics of participants relevant to the evaluation, the number of rounds, and the principal types of problems identified.
Explain how findings influenced item revisions, including important changes to terminology, response categories, or instructions.
A transparent account is more informative than stating that the questionnaire was "pretested and found clear."
Where important problems remain unresolved, their implications for the intended measurement should be acknowledged.
Watch Out
Do not treat cognitive interviewing as a brief satisfaction survey in which participants simply approve or reject questionnaire items. Its methodological value lies in investigating how responses are produced and whether those processes support the intended measurement.
