03 · What You Need to Know
How to Evaluate a Qualitative Interview Guide Before Fieldwork
What Is Pilot Testing a Qualitative Interview Guide?
Pilot testing involves trying out an interview guide and its associated procedures before the main data collection, usually with individuals who can provide relevant feedback about the interview experience.
The purpose is to examine how the questions work in practice, whether participants understand them, and whether the interview process generates material relevant to the research purpose.
For example, a researcher studying teachers' experiences of generative AI may discover that participants interpret the phrase "AI integration" differently. Some may discuss instructional design, others automated assessment, and still others administrative productivity.
A pilot can help the researcher determine whether the question needs clarification or whether those different interpretations are themselves analytically valuable.
The objective is not necessarily to make every participant understand every question identically. In many qualitative approaches, differences in meaning are precisely what the research seeks to understand.
How Is an Interview Pilot Different From a Questionnaire Pretest?
Questionnaire pretesting often examines whether respondents interpret standardized items and response categories consistently enough to support the intended measurement.
Qualitative interview piloting has a different emphasis. It examines whether questions invite relevant experiences, interpretations, and accounts while allowing participants to introduce meanings the researcher did not anticipate.
Questionnaire pretesting
Typically evaluates item comprehension, response options, administration, and the suitability of standardized measurement procedures.
Qualitative interview piloting
Evaluates whether an interview guide and interviewing process support meaningful, sufficiently open, and methodologically appropriate qualitative inquiry.
Both activities may examine clarity and participant burden, but they should not be evaluated using identical assumptions.
For the related measurement issues, see Should You Pilot Test Research Questionnaires Before the Main Study?.
When Is an Interview Pilot Particularly Valuable?
Piloting may be especially useful when the interviewer has limited experience, the research topic is sensitive, the participant population is unfamiliar, or the interview guide contains abstract concepts.
It may also be valuable when interviews are conducted remotely, through an interpreter, across languages, or within relationships involving unequal authority.
For example, a researcher interviewing students about difficulties with their own university may need to examine whether questions encourage candid discussion without implying that particular answers are expected.
A pilot may reveal that participants hesitate to criticize institutional practices or assume that the researcher represents university management.
Such findings may require changes to the interview introduction, confidentiality explanation, recruitment arrangements, or questioning approach.
When Might a Separate Pilot Be Unnecessary?
A separate pilot may be unnecessary when the interview guide has already undergone suitable review, the interviewer is experienced, the procedures are familiar, and the methodology explicitly supports iterative refinement during data collection.
However, familiarity does not eliminate every source of uncertainty.
An experienced interviewer entering a new cultural or institutional setting may still benefit from preliminary conversations about terminology, accessibility, or participant expectations.
The decision should therefore be based on the likely contribution of testing rather than a universal requirement to conduct a fixed number of pilot interviews.
What Should Be Examined During the Pilot?
| Component |
What to examine |
Possible problem |
| Opening questions |
Whether participants can begin comfortably and understand the topic |
Questions feel abrupt or intimidating. |
| Question wording |
Whether participants understand the intended inquiry |
Unfamiliar or ambiguous terminology |
| Question openness |
Whether prompts invite description and reflection |
Leading questions or unnecessary yes-or-no responses |
| Question sequence |
Whether the discussion develops coherently |
Sensitive or complex questions appear too early. |
| Probing |
Whether follow-up prompts help clarify and deepen accounts |
Probes impose assumptions or interrupt participants. |
| Duration |
Whether the interview fits the available time without undue pressure |
Important topics are rushed or omitted. |
| Interviewer practice |
Whether the interviewer listens, follows up, and manages transitions appropriately |
Excessive talking or premature interpretation |
| Recording and logistics |
Whether approved recording, communication, and note-taking procedures function |
Audio loss, interruptions, or privacy difficulties |
The selected evaluation criteria should reflect the intended qualitative methodology. A narrative interview, for instance, may require a different approach to sequencing and interruption from a structured qualitative interview.
How Do You Know Whether an Interview Question Is Clear?
Clarity concerns whether participants can understand what is being asked without unnecessary confusion.
One approach is to ask pilot participants how they interpreted particular questions after the interview.
For example, the question "How do you experience institutional support?" may be understandable to a researcher but abstract to some students.
A more accessible prompt might ask participants to describe a recent occasion when they needed assistance from the university.
However, clarity does not always require narrowing the question. Broad prompts may be appropriate when researchers wish to understand how participants themselves define a concept.
The important distinction is between productive openness and confusion that prevents participants from responding meaningfully.
Should Interview Questions Be Open-Ended?
Open-ended questions are commonly appropriate for qualitative interviews because they allow participants to describe experiences in their own terms.
For example, "Can you describe how you decide whether to use generative AI when preparing assessments?" may produce more detailed material than "Do you use generative AI for assessment?"
However, closed questions are not universally prohibited. They may establish background information, clarify an event, or provide a transition.
The methodological issue is whether the overall interview enables the depth and openness required by the research question.
A pilot can reveal when an apparently open question nevertheless encourages a narrow or predictable response.
How Can You Identify Leading Questions?
A leading question implies a preferred answer or embeds an assumption that participants may not share.
For example, "How has generative AI improved your teaching efficiency?" assumes that improvement occurred.
A less prescriptive alternative is "What changes, if any, have you experienced in your teaching work since using generative AI?"
The second formulation permits positive, negative, mixed, or absent changes.
During piloting, researchers should examine whether participants appear to accept assumptions introduced by the interviewer rather than describing their own experiences.
However, removing leading wording does not eliminate all interviewer influence. Tone, reactions, follow-up questions, and institutional relationships may also shape responses.
Should You Test the Sequence of Interview Questions?
Yes, when sequence may affect comfort, comprehension, or the development of the conversation.
Interviews often begin with accessible contextual questions before moving toward more complex or sensitive topics.
For example, a study of teachers' experiences with AI-related academic integrity concerns might begin with general instructional practices before exploring specific incidents or institutional conflicts.
A pilot may reveal that participants need more context before answering an abstract question or that repeated transitions disrupt the narrative.
However, qualitative interviews do not always require a fixed sequence. Some approaches prioritize following the participant's account rather than maintaining a predetermined order.
The guide should support the interviewer's judgment without unnecessarily constraining it.
How Should Probing Questions Be Evaluated?
Probes are follow-up prompts that invite participants to elaborate, clarify, or provide examples.
Examples include "Can you tell me more about that?", "What happened next?", and "How did you understand that situation?"
A pilot can help researchers identify where probes are needed and whether particular follow-ups introduce assumptions.
For example, asking "Was that frustrating?" may suggest an emotion that the participant has not expressed. Asking "How did you feel about that experience?" leaves more room for the participant's interpretation.
Probing should deepen understanding without turning the interview into a search for predetermined themes.
Should You Test Whether Questions Produce Rich Data?
Yes, but richness should not be equated with interview length or the number of words spoken.
A lengthy response may contain little information relevant to the research question, while a concise account may illuminate a consequential experience.
Researchers should examine whether the pilot interview provides opportunities for participants to describe processes, meanings, circumstances, and variation relevant to the chosen methodology.
For example, if participants repeatedly provide general opinions when the research seeks detailed accounts of decision-making, the questions may need to invite concrete incidents.
However, researchers should avoid treating the pilot as a test of whether participants will confirm anticipated themes.
How Does the Planned Qualitative Analysis Affect Piloting?
The analytic approach influences what kinds of accounts the interview guide needs to support.
A phenomenological inquiry may emphasize descriptions of lived experience, while narrative inquiry may require attention to chronology and how participants construct accounts over time.
Grounded theory research may involve questions about processes, actions, conditions, and consequences, with subsequent interviews informed by emerging analysis.
Reflexive thematic analysis may accommodate a wide range of interview structures, depending on the research question and theoretical orientation.
Piloting should therefore examine whether the questions are suitable for the intended analysis rather than applying one universal standard of interview quality.
See How Should Your Planned Qualitative Analysis Inform Interview Questions?.
Should You Pilot Sensitive Interview Questions?
Particular care is warranted when questions concern distressing experiences, professional misconduct, discrimination, academic failure, or other sensitive matters.
A pilot may help identify wording that feels accusatory, unnecessarily intrusive, or difficult to answer.
Researchers should establish appropriate safeguards before conducting the pilot, including confidentiality procedures, voluntary participation, and responses to possible distress.
Testing should not expose participants to avoidable harm merely to determine whether a question is uncomfortable.
Where the topic involves substantial risk, researchers should consult the responsible ethics authority and relevant methodological expertise.
Can Pilot Interviews Reveal Problems With Researcher Positionality?
They may reveal how the interviewer's role, background, assumptions, or institutional relationships influence the interaction.
For example, students interviewed by a university administrator may respond differently from students interviewed by an independent researcher.
A pilot may reveal hesitation, deference, or assumptions about how the information will be used.
Researchers should consider these observations reflexively rather than assuming that a revised question alone can remove power differences.
Reflexive notes following the interview can document how the interviewer contributed to the production of the account.
Should You Record Pilot Interviews?
Recording may be useful when permitted by consent and applicable ethics approval.
Reviewing an audio or video recording can help researchers examine question delivery, interruptions, probing, and the quality of captured data.
However, recording is not universally necessary or appropriate.
Some settings may require alternative documentation because of confidentiality, participant preferences, or practical constraints.
Researchers should use recording procedures consistent with the main study's intended methods and protect pilot data according to approved arrangements.
How Many Pilot Interviews Should You Conduct?
There is no universal number of pilot interviews.
The appropriate scope depends on the complexity of the guide, the researcher's experience, the diversity of participants, and the problems the pilot is intended to investigate.
One interview may reveal an obvious sequencing or wording problem, but it cannot establish that the guide will work equally well across all relevant participant groups.
Additional interviews may be useful when substantial revisions are made or when different populations may interpret questions differently.
Researchers should justify the scope according to the pilot objectives rather than adopting a fixed numerical convention.
Should Pilot Participants Resemble the Main Participants?
Generally, participants should be sufficiently familiar with the topic and context to provide meaningful feedback about the interview process.
For example, an interview guide intended for university teachers implementing AI-supported assessment should ideally be tested with individuals who understand relevant instructional practices.
However, exact equivalence with the main sample is not always possible or necessary.
Colleagues may help identify obvious problems, but their feedback should not be assumed to represent the experiences of intended participants.
Researchers should distinguish expert review, simulated interviews, and pilot interviews with relevant participants.
Can Pilot Interviews Be Included in the Main Dataset?
Sometimes, depending on the design, consent arrangements, methodological compatibility, and changes made after piloting.
If the interview guide changes substantially, pilot accounts may have addressed different topics or provided different opportunities for participants to elaborate.
However, qualitative research does not always require identical questions across interviews. Some approaches explicitly support iterative development.
The decision should therefore follow the qualitative methodology and the analytic role of the pilot material.
Researchers should not automatically include pilot interviews merely to increase the sample size.
See Can You Include Pilot Study Participants or Data in the Main Study?.
Can the Interview Guide Change After Main Data Collection Begins?
Yes. Many qualitative methodologies permit, and sometimes require, responsiveness to emerging understanding.
Researchers may refine prompts, add follow-up questions, or reconsider the sequence as fieldwork progresses.
However, consequential changes should be documented, and modifications affecting approved procedures or participant risks may require ethics review.
Researchers should distinguish legitimate iterative refinement from changing questions solely to obtain preferred findings.
Pilot testing can improve initial readiness without eliminating the need for methodological judgment during the main study.
How Should Pilot Interview Findings Be Documented?
Researchers should record which guide version was tested, the purpose of the pilot, relevant participant characteristics, problems identified, and resulting revisions.
A revision matrix can connect each issue to the decision made.
Reflexive notes may document interviewer assumptions, interactional difficulties, and emerging questions about the research design.
The final methodology should accurately describe whether pilot interviews were excluded, incorporated, or used only to refine procedures.
Researchers should avoid claiming that the interview guide was validated simply because a few pilot participants completed the interview.
Watch Out
Do not evaluate a qualitative interview guide solely by whether participants provide answers matching anticipated categories. An interview that reveals unexpected meanings or challenges the researcher's assumptions may be functioning particularly well.
