Should Qualitative Findings Help Develop a Quantitative Instrument or Intervention?

Should You Explore a Phenomenon Before Trying to Measure or Change It?

You want to develop a questionnaire measuring teachers' responsible use of generative AI. Existing instruments do not adequately represent the practices you intend to investigate, so you are considering interviewing teachers before writing the survey items.

Alternatively, you may be designing an educational intervention and want to understand participants' experiences, needs, and institutional constraints before deciding what the intervention should contain.

Both situations raise a methodological question: should qualitative findings guide the development of the quantitative instrument or intervention, or would this introduce an unnecessary phase into the study?

The answer depends on what is already known, what remains conceptually uncertain, and whether qualitative inquiry can make a substantive contribution to development. Interviews should not become a ceremonial preliminary activity conducted simply to justify calling a study mixed methods.

Use Qualitative Findings When Development Requires Contextual or Conceptual Understanding

How Qualitative Findings Can Inform Quantitative Development

Why Might Qualitative Inquiry Be Necessary Before Measurement?

Quantitative measurement requires researchers to specify what they intend to measure and how observable responses will represent the underlying construct. When a construct is insufficiently defined, premature item development can produce measures that overlook important dimensions or impose assumptions that do not reflect the target population.

Qualitative inquiry can help investigate how participants understand a phenomenon, which practices they consider relevant, what distinctions they make, and how contextual conditions influence their experiences.

For example, researchers developing a measure of responsible generative AI use in assessment might discover that teachers distinguish between using AI to generate assessment materials, evaluate student work, and provide feedback. These distinctions could inform the conceptual framework and candidate items.

However, qualitative findings should be considered alongside relevant theory, prior empirical literature, and established measurement frameworks. Participant accounts are an important source of evidence, but they do not automatically define the complete construct.

When Is Qualitative-First Development Appropriate?

Qualitative-first development may be particularly useful when a phenomenon is emerging, existing measures have questionable relevance to the intended population, or an intervention must respond to poorly understood needs.

It can also be appropriate when researchers are adapting an established instrument to a substantially different cultural, institutional, linguistic, or professional context.

Nevertheless, adaptation does not always require a full exploratory qualitative study. Existing evidence, expert assessment, translation procedures, and cognitive interviews may sometimes provide a more proportionate approach.

The key consideration is whether qualitative evidence will resolve a genuine developmental uncertainty. If the construct is already well established and an appropriate instrument exists, developing another measure may add little value.

How Does This Fit an Exploratory Sequential Mixed-Methods Design?

In an exploratory sequential arrangement, qualitative inquiry precedes and informs a quantitative component. The first phase may establish relevant concepts, generate candidate items, or identify intervention features. The second phase then examines a quantitative research question using the resulting instrument or intervention.

The relationship between the phases is developmental. Qualitative findings are not merely reported before the quantitative results; they influence what is subsequently measured or implemented.

For example, interviews about students' experiences with AI-supported feedback might inform the development of a questionnaire measuring perceived feedback usefulness. The questionnaire could then be evaluated and administered to examine patterns in a broader population.

This is a potential application of an exploratory sequential mixed-methods design.

Importantly, the design should not be identified solely from the order of activities. Researchers must demonstrate how qualitative findings informed the subsequent quantitative procedures and how the phases contribute to the overarching research purpose.

How Do You Translate Qualitative Findings Into Survey Items?

Translating qualitative findings into questionnaire items requires several distinct decisions. A theme identified through interviews is not automatically a valid measurement dimension, and a participant quotation is not necessarily a suitable survey item.

Researchers should first establish a conceptual definition of the intended construct. Qualitative findings can then inform candidate dimensions and the behaviors, perceptions, or experiences that may represent them.

Item wording should be developed with attention to clarity, specificity, response processes, and the intended interpretation of scores.

A defensible process might involve the following activities:

The COSMIN methodology for evaluating measurement instruments emphasizes content validity, including relevance, comprehensiveness, and comprehensibility. Although COSMIN was developed primarily for health measurement instruments, these principles can inform measurement development in other disciplines when applied appropriately.

Researchers should distinguish the generation of candidate content from the subsequent evaluation of measurement quality.

Does Qualitative Evidence Establish Content Validity?

Qualitative evidence can contribute to content validity, but it does not establish content validity automatically.

Interviews may help identify important dimensions and reveal whether proposed items reflect participants' experiences. Cognitive interviews may indicate whether respondents understand questions as intended.

Nevertheless, content validity concerns whether the instrument's content adequately represents the intended construct for its specified population and purpose. This judgment may require evidence from theory, relevant literature, subject-matter experts, and members of the target population.

Furthermore, content validity is only one aspect of instrument evaluation. Evidence concerning internal structure, reliability, relationships with other variables, and response processes may also be necessary, depending on the instrument and intended interpretation.

The Standards for Educational and Psychological Testing emphasize that validity concerns evidence and theory supporting interpretations of test scores for their intended uses, rather than a permanent property conferred on an instrument by a single development procedure.

Can Qualitative Findings Help Develop an Intervention?

Yes. Qualitative inquiry can inform intervention development by identifying participant needs, contextual conditions, implementation barriers, and the ways intended users understand the problem being addressed.

For example, a researcher developing professional development activities for teachers using generative AI might interview teachers about assessment practices, institutional expectations, and perceived risks.

The findings could inform intervention content, delivery arrangements, examples, and implementation supports.

However, an intervention should not be constructed solely from participant preferences. Development should also consider relevant theory, existing evidence, feasibility, ethical requirements, and the intended mechanisms of action.

The UK Medical Research Council and National Institute for Health and Care Research framework for complex interventions emphasizes development, feasibility, evaluation, and implementation as interconnected activities. It also recognizes the importance of context, stakeholder engagement, and program theory.

Qualitative inquiry may contribute to several of these activities, but its role should be specified rather than assumed.

How Is Instrument Development Different From Intervention Development?

Although both may begin with qualitative inquiry, their intended products and evaluation requirements differ.

A questionnaire developed from interviews may require psychometric evaluation. An educational intervention developed from interviews may require feasibility testing and an appropriate outcome evaluation.

Neither product becomes scientifically established merely because its initial design was informed by qualitative evidence.

How Should the Qualitative and Quantitative Phases Be Integrated?

Integration should make the developmental relationship between phases explicit.

Researchers can document how qualitative themes informed construct definitions, candidate items, intervention components, or subsequent research questions.

A development matrix may connect each qualitative finding with the corresponding design decision and its justification.

For example, an interview theme concerning uncertainty about acceptable AI assistance might inform a questionnaire dimension about perceived institutional clarity. The researcher should explain why that dimension follows from the evidence and how the resulting items were evaluated.

This form of integration is commonly described as building, in which findings from one component inform the procedures of another.

The researcher should also specify where integration occurs and how the developmental decisions contribute to the overarching interpretation.

What If the Qualitative Findings Do Not Support the Expected Instrument or Intervention?

Qualitative findings may reveal that the original construct is poorly defined, that intended users interpret it differently, or that the proposed intervention does not address the most consequential needs.

Such findings can justify revising the conceptual framework, narrowing the research objective, or reconsidering whether the proposed instrument or intervention should be developed at all.

Researchers should not force qualitative findings into predetermined dimensions merely to preserve the original development plan.

When substantial changes are made, the rationale and implications should be documented. Depending on the nature of the changes, additional ethical review, stakeholder consultation, or methodological evaluation may be necessary.

When Is a Full Mixed-Methods Study Unnecessary?

Not every qualitative contribution to development requires a mixed-methods design.

For example, a quantitative researcher might conduct a small number of cognitive interviews to improve questionnaire wording before a survey. These interviews may be part of instrument pretesting rather than a distinct qualitative research component contributing to an overarching mixed-methods inference.

Similarly, a study that conducts interviews and produces an intervention prototype without a subsequent quantitative component may be better described as qualitative research or qualitative-informed intervention development.

The distinction depends on the research purpose, methodological treatment of the evidence, and the relationship between components.

Researchers should therefore ask whether their project genuinely requires a mixed-methods research question or whether a more focused developmental approach would be sufficient.

Developing a Questionnaire From Teachers' Experiences With Generative AI

Suppose the interviews suggest that teachers distinguish between disclosure of AI assistance, verification of AI-generated content, and accountability for assessment decisions. These could become candidate dimensions, but they should not automatically be treated as established factors.

Subsequent quantitative analysis might support the proposed structure, suggest revisions, or reveal that some dimensions are not empirically distinguishable in the intended population.

The contribution of the mixed-methods design lies in the documented relationship between conceptual exploration and measurement evaluation. The resulting instrument's usefulness depends on the quality of both phases and the evidence supporting its intended interpretations.

Common Mistakes in Qualitative-Informed Instrument and Intervention Development

How to Decide Whether Qualitative Development Is Necessary

Begin by examining the existing conceptual and empirical literature. Determine whether the intended construct or intervention is already sufficiently understood and whether suitable instruments or established intervention components are available.

If important conceptual or contextual gaps remain, identify what qualitative inquiry could resolve and how its findings would influence subsequent development.

Plan development and evaluation as distinct activities. For instruments, specify what measurement properties will be investigated. For interventions, distinguish feasibility, implementation, and outcome evaluation.

Above all, preserve a transparent record of how qualitative evidence influenced the final product. The credibility of the developmental relationship depends on showing what was learned, what decisions followed, and how the resulting instrument or intervention was subsequently evaluated.

Check Whether Qualitative Findings Should Inform Development

Frequently Asked Questions About Qualitative-Informed Development

Use Qualitative Evidence to Inform Development, Not Replace Evaluation

How to Cite This Guide

This guide is intended to be read, shared, and used in research, teaching, and academic work. If you draw on its ideas, explanations, or other content, please acknowledge the source by citing the guide. Doing so gives appropriate credit and helps your readers locate the original resource.