How Do Researchers Preserve Their Substantive Judgment When AI Helps Design Qualitative Questions?

When AI Helps Write Your Interview Questions, Who Is Actually Designing the Inquiry?

You have developed a qualitative research proposal and want to prepare an interview guide. You provide your research questions to ChatGPT and ask it to generate open-ended questions, suggest follow-up probes, and organize the discussion.

The resulting guide looks methodologically sound. Its questions are clear, its sequence appears logical, and its suggested probes seem relevant. You accept most of the recommendations because they are difficult to fault on the surface.

But consider what may have happened during that process. AI may have decided which experiences deserve attention, which concepts should frame the discussion, and which possible explanations should be explored. It may have introduced familiar categories such as benefits, challenges, barriers, and opportunities, even when those categories were not central to your research questions.

You remain the person conducting the study, but some of its substantive design decisions may have shifted toward the AI system without being recognized as decisions.

How can researchers benefit from AI assistance while preserving responsibility for what their qualitative questions investigate, assume, and leave open?

Preserve Judgment by Making Methodological Decisions Explicit

What Substantive Researcher Judgment Means in AI-Assisted Qualitative Question Design

Researcher Judgment Is Not the Same as Checking AI Output

When researchers use AI to develop interview questions, they may interpret their responsibility as reviewing the generated material for clarity, grammar, and relevance.

These checks are useful, but they address only part of the methodological problem.

Substantive judgment concerns the decisions that shape the inquiry itself. Researchers determine which experiences are relevant, how theoretical concepts should inform questioning, what assumptions need examination, and how participants can contribute interpretations that differ from those anticipated.

For example, suppose AI generates the question: "What challenges have instructors experienced when adopting generative AI?"

The question is understandable and open-ended. However, accepting it also means accepting adoption and challenges as organizing concepts. That framing may be inappropriate if the study investigates how instructors decide whether to use AI at all.

The researcher must therefore evaluate not only how the question is written but also what it presupposes and what it makes possible to investigate.

AI Can Influence the Inquiry Before Data Collection Begins

Qualitative research is often associated with openness to participants' perspectives and meanings. However, the questions used during data collection inevitably shape the conversations that follow.

Generative AI can influence this process by suggesting familiar categories, emphasizing particular aspects of a phenomenon, or presenting one conceptual framing as though it were the natural way to investigate the topic.

For example, a researcher studying university instructors' experiences with generative AI might receive questions about perceived usefulness, ease of use, adoption barriers, and behavioral intention.

These concepts may be appropriate for a study explicitly informed by technology acceptance theory. They are not automatically suitable for a phenomenological inquiry into how instructors experience professional responsibility when using AI.

The problem is not that AI-generated theoretical concepts are necessarily incorrect. It is that they may be incorporated without examining whether they belong to the study's methodological orientation.

Researchers should recognize that accepting an AI-generated question can also mean accepting an implicit decision about what matters.

Distinguish Methodological Assistance From Methodological Authority

AI can help researchers perform tasks involved in question development. However, performing a task is different from having the authority to determine whether the task is appropriate.

The distinction is not intended to exclude AI from consequential stages. Rather, it ensures that AI suggestions remain subject to explicit methodological reasoning.

Why a Plausible Question Can Still Be Methodologically Inappropriate

One difficulty with AI-generated interview questions is that they can sound convincing even when they do not adequately serve the research inquiry.

Consider the question: "How has generative AI transformed your professional identity as a teacher?"

The wording may appear sophisticated and relevant to educational technology research. However, it assumes that AI has transformed professional identity and invites participants to explain a change that may not have occurred.

A more open question might ask: "Have your experiences with generative AI influenced how you think about your professional role, if at all?"

Alternatively, if professional identity is not central to the study, the researcher may decide that neither question belongs in the guide.

This final possibility is important. Preserving judgment does not always mean improving an AI-generated question. Sometimes it means rejecting the question entirely.

Maintain Alignment With the Qualitative Methodological Approach

Qualitative methodologies differ in their purposes, epistemological assumptions, and approaches to data generation and interpretation.

A phenomenological study may seek detailed accounts of lived experience and meaning. A grounded theory study may investigate actions, interactions, and processes relevant to developing theoretical explanations. A case study may examine a phenomenon within a bounded context.

These descriptions are broad, and each tradition contains important methodological variations.

AI may produce an apparently suitable set of open-ended questions without recognizing the distinctions among these approaches.

For example, questions about barriers to technology adoption may fit one theoretical inquiry but impose an inappropriate framework on another.

Researchers should therefore specify their methodological orientation and evaluate whether each question contributes to the kind of knowledge the study seeks to develop.

Kallio and colleagues (2016) emphasize the importance of identifying methodological prerequisites and using relevant prior knowledge before formulating and piloting a semi-structured interview guide.

Use Reflexivity to Examine AI's Influence on Your Decisions

Reflexivity involves critically examining how the researcher's assumptions, position, experiences, and decisions shape the research process.

When AI participates in question development, researchers should also examine how its suggestions influence their own understanding of the phenomenon.

For example, after receiving an AI-generated guide organized around benefits and challenges, a researcher might begin to assume that participants' experiences should fit those categories.

That assumption could influence not only the interview questions but also subsequent probing and interpretation.

Braun and Clarke (2022) emphasize the active and situated role of researchers in qualitative analysis. Although their discussion concerns reflexive thematic analysis rather than AI-assisted interview guide development specifically, the broader principle is relevant: qualitative research decisions are not neutral operations performed independently of the researcher.

AI-generated suggestions should therefore become material for reflexive consideration rather than a substitute for it.

Keep a Record of Why Questions Were Accepted or Rejected

One practical way to preserve researcher judgment is to document consequential decisions during interview guide development.

For each important AI-generated suggestion, the researcher can record the proposed question, its intended contribution, any methodological concern, and the reason for accepting, revising, or rejecting it.

This record does not need to become an elaborate administrative exercise. Its purpose is to make substantive decisions traceable rather than allowing them to disappear behind a polished final guide.

Documentation can also support methodological transparency when explaining how AI contributed to the study.

How Should You Prompt AI Without Delegating the Researcher's Judgment?

A useful prompt should request alternatives, expose assumptions, and invite critical comparison rather than asking AI to determine the single best question without justification.

This prompt encourages AI to make its suggestions open to examination. It does not guarantee that the model will identify every conceptual problem.

The researcher should still compare the output with the study's methodological literature and research context.

Do Not Allow AI to Predetermine the Themes

Generative AI may suggest interview questions organized around themes it expects to be important, such as efficiency, trust, ethical concerns, institutional support, or resistance.

These may be reasonable areas to explore when justified by theory or prior research. However, their inclusion should not be mistaken for evidence that they will emerge from participants' accounts.

For example, a guide that repeatedly asks about AI-related anxiety may encourage participants to discuss anxiety even when it is not central to their experiences.

Researchers should distinguish sensitizing concepts that inform the inquiry from findings that can only be established through data collection and analysis.

Some qualitative approaches legitimately use theory-informed questions. The concern is not the presence of prior concepts but the unexamined assumption that participants' accounts must conform to them.

AI can assist in generating open-ended interview questions, but researchers must preserve the possibility that participants will introduce meanings and experiences outside the anticipated categories.

Recognize When AI Makes the Guide Too Generic

AI-generated questions may be so broadly applicable that they fail to investigate the distinctive circumstances of the study.

For example, "What challenges do you face with generative AI?" could be asked in almost any professional setting.

If the study concerns instructors' decisions under institutional expectations, the guide should provide opportunities to explore how those expectations are encountered, interpreted, or negotiated.

However, contextual specificity should not be achieved by inventing policies or assuming that particular conflicts occurred.

Researchers should evaluate whether the questions are grounded in verified contextual information and remain open to contrasting accounts.

The risk of generic AI-generated interview guides illustrates why apparently reasonable questions still require substantive review.

Question Wording Can Quietly Change the Research Focus

AI may suggest revisions that improve readability while altering what the question investigates.

For example, the question "Can you describe how you decided whether to use AI for assessment?" concerns decision-making.

A suggested revision such as "How satisfied are you with using AI for assessment?" shifts the focus toward satisfaction.

Both questions may be relevant to educational technology research, but they are not interchangeable.

Researchers should examine whether revisions preserve the original inquiry rather than evaluating them solely for grammatical quality or conversational appeal.

Similar concerns arise when AI helps detect leading questions or sequence interview questions.

A technically smoother question is not necessarily a methodologically better one.

Preserve Judgment During Interview Probing

Researcher judgment remains important after the guide has been finalized.

During an interview, participants may introduce experiences that were not anticipated during question development. The interviewer must decide whether to pursue those accounts, ask for clarification, or return to the planned topics.

AI-generated probes may provide useful preparation, but they cannot determine which follow-up is appropriate before hearing the participant's actual response.

For example, if an instructor says that using AI made them feel less independent, the researcher should not immediately assume that the participant became dependent on AI-generated outputs.

The expression might instead refer to institutional restrictions, professional expectations, or another experience.

Asking what the participant means by "less independent" preserves the opportunity for the participant to define the experience.

Researchers can use AI to prepare possible probes, but attentive listening and contextual interpretation remain central to interviewing.

Do Not Confuse AI Review With Independent Evaluation

After generating an interview guide, researchers may ask the same AI system to assess whether the questions are appropriate.

The model may identify genuine problems, but its review does not establish how participants will understand the questions or whether the guide will function effectively during actual interviews.

It may also reproduce assumptions introduced during the original generation process.

Independent methodological feedback and pilot interviewing provide different forms of evaluation.

Castillo-Montoya's (2016) Interview Protocol Refinement framework emphasizes alignment, conversational design, feedback, and piloting.

Researchers should therefore distinguish AI-assisted preliminary review from pilot testing with actual participants.

Protect Participant Information When Seeking AI Assistance

AI assistance may continue after data collection begins, particularly when researchers want help developing follow-up questions or reviewing interview transcripts.

However, actual participant statements may contain personal information, identifiable institutional circumstances, or sensitive disclosures.

Researchers should not upload such material to external AI systems without appropriate authorization and safeguards consistent with the approved research protocol, participant consent, and applicable data-protection requirements.

When possible, researchers can use generalized or hypothetical examples to obtain preliminary methodological suggestions without exposing identifiable accounts.

Ethical judgment includes deciding not only what questions to ask participants but also what information may appropriately be shared with AI tools.

What Should Researchers Report About AI-Assisted Question Design?

When AI materially contributes to interview guide development, researchers should describe its role accurately and follow applicable institutional, journal, and funder requirements.

For example, a methodology section may explain that AI was used to generate candidate questions and identify potential wording problems, followed by researcher revision, methodological feedback, and pilot interviewing.

Researchers should not claim that the guide was independently validated merely because AI reviewed it favorably.

Nor should they imply that the model conducted interviews, obtained participant evidence, or established the appropriateness of the qualitative design when those activities did not occur.

Transparent reporting should distinguish AI-generated suggestions from the substantive decisions made by the researcher and the evidence obtained through actual research procedures.

Researcher Judgment Does Not Mean Rejecting Every AI Suggestion

Preserving substantive judgment should not be confused with minimizing AI use at all costs.

AI may identify an assumption the researcher overlooked, suggest a more accessible question, or reveal that several questions investigate the same issue.

Accepting such suggestions can be methodologically appropriate when the researcher understands and can justify the decision.

Conversely, rejecting a suggestion simply because it originated from AI does not demonstrate rigor.

The relevant distinction is between critically evaluated assistance and unexamined delegation.

A researcher who can explain why a question was retained, revised, or removed exercises more substantive control than one who accepts or rejects AI output without examining its implications.

Preserving Researcher Judgment While Developing an AI-Assisted Interview Guide

The researcher did not preserve judgment merely by editing the AI-generated wording. The more important decision was recognizing that the model had introduced a technology acceptance framing that was not necessarily appropriate to the study.

AI remained useful during question development, but the researcher determined the conceptual boundaries of the inquiry.

Common Misconceptions About Researcher Judgment and AI-Assisted Qualitative Questions

Keep the Researcher Responsible for What the Questions Make Possible

When using AI to develop qualitative questions, begin with the research problem and methodological orientation rather than asking the model to define the inquiry from a broad topic.

Use AI to generate alternatives, identify possible assumptions, and examine question wording. Then determine whether each suggestion contributes to the study and whether it preserves opportunities for participants to introduce unexpected experiences.

A useful practical test is whether you can explain why each main question belongs in the guide without relying on the AI system's explanation.

If you cannot justify what a question investigates, why it is appropriate for the participants, or how it contributes to the research inquiry, the question requires further consideration regardless of how professionally it is written.

Before Finalizing AI-Assisted Qualitative Interview Questions

Frequently Asked Questions About Researcher Judgment in AI-Assisted Qualitative Research

AI Can Suggest the Questions, but the Researcher Must Decide What Is Worth Asking

How to Cite This Guide

This guide is intended to be read, shared, and used in research, teaching, and academic work. If you draw on its ideas, explanations, or other content, please acknowledge the source by citing the guide. Doing so gives appropriate credit and helps your readers locate the original resource.