03 · What You Need to Know
How to Pilot Test a Research Questionnaire Properly
What Is Questionnaire Pilot Testing?
Questionnaire pilot testing involves administering a proposed questionnaire under preliminary conditions to identify problems before its use in the main study.
Depending on the purpose, researchers may assess item interpretation, response processes, questionnaire navigation, administration time, missing responses, or data management.
For example, a researcher developing a questionnaire about teachers' use of generative AI may discover that respondents interpret "AI use" differently. Some may include grammar correction tools, while others consider only generative systems that produce instructional content.
Such differences may undermine the interpretation of responses if the intended construct is not clearly defined.
Pretesting can identify these issues before the instrument is administered to a larger sample.
How Is Questionnaire Pretesting Different From a Pilot Study?
Questionnaire pretesting focuses on the instrument and its response processes.
A pilot study may rehearse a broader set of research procedures, including recruitment, consent, questionnaire administration, follow-up, and data processing.
Questionnaire pretest
Examines whether questionnaire items, instructions, response options, and administration procedures work as intended.
Broader pilot study
Rehearses selected or multiple components of the proposed definitive study, potentially including questionnaire administration within the full research workflow.
The terms are sometimes used loosely, but researchers should describe the actual procedures rather than rely on the label alone.
For a broader comparison, see Pretest vs. Pilot Study vs. Feasibility Study: What's the Difference?.
When Is Questionnaire Pretesting Especially Important?
Pretesting is particularly important when an instrument is newly developed, substantially revised, translated, adapted to a different population, or administered using unfamiliar technology.
It may also be valuable when the questionnaire contains complex instructions, sensitive questions, difficult recall tasks, or elaborate skip patterns.
For example, a questionnaire originally developed for face-to-face university teaching may not fully capture instructional practices in asynchronous online education.
Similarly, an instrument developed before widespread adoption of generative AI may contain terminology that respondents now interpret differently.
The need for pretesting depends on whether existing evidence adequately supports the instrument's intended use in the new context.
What Should Be Examined During Questionnaire Pretesting?
| Component |
What to examine |
Potential problem |
| Item wording |
Whether respondents interpret questions as intended |
Ambiguous or unfamiliar terminology |
| Instructions |
Whether participants understand what they should do |
Incorrect response procedures |
| Response options |
Whether available choices represent plausible answers |
Missing, overlapping, or unsuitable categories |
| Recall requirements |
Whether participants can retrieve the requested information |
Unreliable estimates or guesses |
| Questionnaire order |
Whether sequence affects understanding or response behavior |
Context effects or confusing transitions |
| Skip logic |
Whether conditional questions appear correctly |
Participants receive irrelevant questions or miss required items |
| Administration time |
Whether the questionnaire can be completed within a reasonable period |
Excessive burden or incomplete responses |
| Accessibility |
Whether the questionnaire is usable by the intended population |
Device, language, or disability-related barriers |
| Data capture |
Whether responses are stored and exported correctly |
Missing, duplicated, or incorrectly coded records |
Not every instrument requires the same testing intensity. The evaluation should focus on the problems most likely to affect its intended use.
Why Is Cognitive Interviewing Useful?
Cognitive interviewing examines how respondents understand questions, retrieve information, form judgments, and select answers.
Researchers may ask participants to explain what a question means to them or describe how they selected a response.
For example, after answering an item about academic engagement, a participant might be asked what activities they considered when interpreting the term "participation."
The response may reveal that some students consider only live online meetings, while others include discussion forums and asynchronous learning tasks.
These differences may indicate that the item requires clarification.
Willis provides an influential methodological treatment of cognitive interviewing as a tool for questionnaire development and evaluation.
What Is the Difference Between Think-Aloud and Verbal Probing?
Think-aloud techniques invite respondents to describe their thinking while answering questionnaire items.
Verbal probing involves follow-up questions about how respondents interpreted or answered particular items.
Both can provide evidence about response processes, but they may influence the answering experience differently.
Some participants find continuous think-aloud tasks difficult or unnatural. Targeted probes may be more suitable for particular items.
Researchers should choose techniques according to the evaluation purpose and avoid assuming that verbal reports reveal every cognitive process accurately.
How Do You Identify Ambiguous Questionnaire Items?
Ambiguity occurs when wording permits interpretations that differ in ways relevant to the intended measurement.
For example, the item "I regularly use AI for teaching" may be interpreted as referring to lesson preparation, automated grading, feedback generation, or classroom demonstrations.
The word "regularly" may also refer to different frequencies among respondents.
Pretesting can reveal whether these differences are consequential.
Researchers may need to define the activity, specify a reference period, or separate distinct behaviors into different items.
However, increasing specificity should not unnecessarily narrow a construct that is intentionally broad.
Should You Test Response Options?
Yes. Response options influence how respondents translate their experiences into recorded answers.
For categorical questions, options should be appropriate to the construct and sufficiently comprehensive for the intended population.
For example, asking teachers how often they use a technology may require a defined reference period and response categories that distinguish meaningful levels of use.
Researchers should examine whether categories overlap, whether relevant answers are missing, and whether participants understand the distinctions.
For rating scales, response labels and the number of categories should be consistent with the intended measurement approach.
Pretesting can reveal response difficulties, but it does not independently establish the psychometric adequacy of a scale.
What About Double-Barreled Questions?
A double-barreled item asks respondents to evaluate more than one issue while providing only one answer.
For example, "The online learning platform is accessible and easy to use" combines accessibility and usability.
A respondent may find the platform accessible but difficult to navigate.
The item would not allow that distinction to be expressed clearly.
Pretesting may reveal the problem when participants explain that they agree with one part but disagree with another.
Researchers should consider separating the concepts when they represent distinct aspects of the intended construct.
Should You Test Recall Periods?
Yes, particularly when questions require respondents to remember events or estimate frequencies.
For example, asking students how many times they contacted academic support during the previous year may produce less reliable recall than asking about a shorter, clearly defined period.
However, shorter reference periods are not universally preferable. The appropriate period depends on the phenomenon and research question.
Pretesting can help determine whether respondents understand the time frame and can retrieve the requested information.
Researchers should avoid assuming that confident answers are necessarily accurate.
How Should Questionnaire Length Be Evaluated?
Administration time and perceived burden should be assessed under conditions resembling the main study.
Researchers may record completion times and ask participants whether instructions, repetitive items, or sensitive questions created difficulties.
However, a short completion time does not necessarily indicate good questionnaire quality.
Respondents may rush, skip items, or select answers without adequate consideration.
Similarly, a longer questionnaire is not automatically inappropriate if the information is necessary and participants can complete it meaningfully.
The objective is to balance measurement requirements with reasonable participant burden.
Should You Pilot Test Online Questionnaire Functionality?
Yes, when the questionnaire is administered electronically.
Researchers should test whether the form displays correctly on relevant devices, whether skip logic operates as intended, and whether responses are recorded accurately.
For example, a conditional question may appear correctly on a desktop browser but fail on a mobile device.
Another problem may occur when exported response categories do not match the coding expected by the analysis software.
Technical testing can often begin with synthetic responses before involving participants.
However, successful technical testing does not establish that respondents understand the questions.
Should You Examine Missing Responses?
Yes. Missing responses may indicate confusing wording, unsuitable response options, sensitive content, technical problems, or participant fatigue.
For example, if many pretest participants omit a question about institutional support, the researcher should investigate why.
However, missingness should not automatically be attributed to poor wording.
Some participants may legitimately decline to answer sensitive questions, while others may not have relevant experience.
Researchers should distinguish these possibilities before revising the instrument.
Can Pilot Testing Establish Questionnaire Validity?
Pilot testing can contribute evidence relevant to validity, particularly concerning item comprehension, response processes, and the suitability of administration procedures.
However, validity concerns the degree to which evidence and theory support the intended interpretation and use of scores.
A questionnaire does not become universally valid simply because a small group completed it successfully.
Depending on the intended measurement, additional evidence may concern content, internal structure, relationships with other variables, and consequences of use.
The Standards for Educational and Psychological Testing provide an authoritative framework for evaluating such evidence.
Researchers should specify what aspect of validity the pretest addresses rather than stating that the questionnaire was validated through pilot testing alone.
Can Pilot Testing Establish Reliability?
A pilot may provide preliminary information about score consistency or stability, but the usefulness of such estimates depends on sample size, measurement design, and intended interpretation.
For example, researchers may calculate Cronbach's alpha for a proposed multi-item scale.
However, alpha does not establish unidimensionality, and a high value does not independently demonstrate that the items measure the intended construct.
Small pilot samples can also produce imprecise reliability estimates.
Researchers should choose reliability evidence appropriate to the measurement model rather than treating one coefficient as a universal requirement.
Is Cronbach's Alpha Required for Every Questionnaire?
No. Cronbach's alpha is relevant to certain multi-item score interpretations but is not appropriate for every questionnaire.
A demographic questionnaire, a set of independent behavioral questions, or a formative index may not require internal consistency estimation in the same way as a reflective multi-item scale.
Researchers should first determine whether the items are intended to represent a common underlying construct.
Even when alpha is relevant, its interpretation depends on assumptions and the characteristics of the items.
Reporting alpha without explaining the measurement model may create a misleading impression of psychometric adequacy.
Should You Conduct Factor Analysis on Pilot Data?
Factor analysis may be relevant when evaluating the internal structure of a multi-item instrument.
However, small questionnaire pilots conducted primarily to assess usability or comprehension may not provide sufficient data for stable factor analysis.
There is no universal sample size rule that guarantees adequate factor analysis. Requirements depend on factors such as communalities, number of factors, item characteristics, and model complexity.
Researchers should distinguish an instrument pretest from a substantive psychometric validation study.
When internal structure is central to the research, an appropriately designed and adequately sized evaluation may be necessary.
How Many Participants Should Be Included in a Questionnaire Pilot?
There is no universal number suitable for every purpose.
A small cognitive interviewing study may identify major comprehension problems, while estimating reliability or evaluating factor structure may require substantially more observations.
The appropriate sample depends on whether the objective is usability testing, item comprehension, response distribution assessment, or psychometric evaluation.
Researchers should avoid treating commonly repeated sample-size rules as universally sufficient.
The sample should also reflect relevant characteristics of the intended population, particularly when language, experience, or accessibility may affect response processes.
Should Pilot Participants Resemble the Main Study Population?
Generally, yes, when the objective is to assess how intended respondents will understand and complete the questionnaire.
For example, a questionnaire intended for first-year online students should not be evaluated exclusively by experienced faculty members.
Experts may identify conceptual or technical problems, but they may not anticipate how students interpret the questions.
Where the population is heterogeneous, pretesting may need to include participants with different relevant characteristics.
However, a small pretest sample cannot establish that every subgroup will interpret the instrument identically.
Should Translated Questionnaires Be Pretested?
Yes, when translation or cultural adaptation may affect item meaning.
Literal translation does not guarantee conceptual equivalence.
For example, terminology concerning teacher autonomy, institutional support, or academic engagement may have different connotations across educational settings.
Pretesting with intended respondents can identify confusing or culturally inappropriate wording.
However, translation pretesting alone does not establish measurement invariance or equivalence of scores across language groups.
Those claims require additional evidence appropriate to the intended comparisons.
Can You Revise Questionnaire Items After the Pilot?
Yes. Identifying and correcting problems is a principal purpose of pretesting.
Researchers may revise wording, instructions, response categories, questionnaire order, or technical functions.
However, substantial revisions may alter the construct being measured or invalidate previous psychometric evidence.
Researchers should document the changes and consider whether further pretesting is necessary.
Where the questionnaire is part of an approved research protocol, consequential changes may require an ethics amendment.
Can Pilot Questionnaire Responses Be Included in the Main Dataset?
Sometimes, but inclusion requires careful consideration.
If the questionnaire changes substantially after the pilot, responses collected using different versions may not be comparable.
Even minor changes can affect response interpretation.
Researchers should consider the study design, consent arrangements, measurement compatibility, and whether inclusion was planned.
Responses should not be combined merely to increase sample size.
For additional considerations, see Can You Include Pilot Study Participants or Data in the Main Study?.
How Should Questionnaire Pretesting Be Documented?
Researchers should record which version was tested, who participated, what procedures were used, and what problems were identified.
Documentation may include cognitive interview findings, completion times, missing response patterns, technical errors, and decisions about item revision.
A revision log can explain why wording or response options changed.
The final research report should distinguish pretesting evidence from formal psychometric evaluation.
Statements such as "the questionnaire was validated through a pilot study" should be avoided unless the evidence genuinely supports the specific validity claims being made.
Watch Out
Do not equate successful questionnaire completion, a high Cronbach's alpha, or favorable participant feedback with comprehensive instrument validity. These observations may contribute useful evidence, but each addresses only particular aspects of questionnaire quality.
