03 · What You Need to Know
Designing Qualitative Inquiry Within an Experimental Evaluation
What Does It Mean to Embed Qualitative Research Within a Trial?
An embedded qualitative component is a planned qualitative investigation conducted within or alongside a broader experimental design. The experiment ordinarily addresses the principal causal or evaluative question, while qualitative inquiry addresses a related question about the intervention or the conditions under which it is studied.
For example, a randomized trial may estimate the effect of an AI-supported feedback intervention on students' writing performance. Embedded interviews may investigate how students interpret feedback, what difficulties they encounter, and how teachers incorporate the system into instruction.
The qualitative component is not simply a collection of participant testimonials. It requires a defensible qualitative research question, sampling strategy, data collection procedures, and analytical approach.
Fetters, Curry, and Creswell (2013) describe embedding as a form of mixed-methods integration in which one methodological component is incorporated within a broader design.
The defining relationship is the supporting contribution of the qualitative inquiry to the primary evaluation.
When Is Embedded Qualitative Inquiry Particularly Useful?
Qualitative inquiry is most useful when the experimental outcome does not provide sufficient information about a consequential aspect of the intervention.
| Evaluation concern |
Qualitative contribution |
Potential implication |
| Implementation fidelity |
Investigate how the intervention was delivered and whether important adaptations occurred. |
Clarifies the intervention actually experienced by participants. |
| Participant acceptability |
Explore perceived appropriateness, burden, and reactions. |
Informs interpretation of uptake and future implementation. |
| Intervention engagement |
Investigate how participants use or avoid intervention activities. |
Provides context for interpreting exposure and participation indicators. |
| Contextual variation |
Examine institutional, organizational, or instructional conditions. |
Identifies conditions potentially relevant to implementation. |
| Possible mechanisms |
Investigate experiences and processes consistent with a proposed program theory. |
Develops or refines plausible explanations without independently proving mediation. |
| Unexpected outcomes |
Explore relevant experiences or implementation differences. |
Generates contextual interpretations and questions for further investigation. |
These purposes can overlap, but they should not be treated as interchangeable. Investigating acceptability is not the same as measuring implementation fidelity, and understanding participants' experiences does not necessarily establish an intervention's causal mechanism.
How Is Embedded Qualitative Research Different From a Process Evaluation?
A process evaluation examines how an intervention is implemented and how relevant processes and contextual conditions may influence its operation.
Qualitative research may contribute substantially to a process evaluation, but process evaluations can also use quantitative indicators, observations, administrative records, and other forms of evidence.
Moore and colleagues (2015), in guidance on process evaluation of complex interventions, distinguish implementation, mechanisms of impact, and context as important areas of investigation.
Consequently, an embedded qualitative component may form part of a broader process evaluation without being equivalent to the entire process evaluation.
Researchers should specify whether their interviews investigate intervention delivery, participant responses, contextual conditions, or another defined issue.
When Should the Qualitative Component Be Planned?
Ideally, its purpose should be considered during trial design, particularly when qualitative evidence is needed to interpret implementation or participant experiences.
Early planning allows researchers to align qualitative questions with the intervention's program theory, establish recruitment procedures, obtain appropriate ethical approval, and identify relevant observation periods.
However, some qualitative questions may emerge from trial findings. For example, an unexpected implementation problem may justify additional qualitative investigation.
Such changes should be documented transparently, including whether the qualitative component was prespecified or developed after reviewing trial results.
Researchers should not retrospectively describe exploratory interviews as though they were part of the original protocol when they were not.
Should Qualitative Data Be Collected Before, During, or After the Trial?
The appropriate timing depends on the qualitative question.
Before the trial, qualitative inquiry may inform intervention refinement, recruitment procedures, or assessment of contextual readiness. During the trial, interviews and observations may investigate implementation, participant engagement, and emerging difficulties. After the trial, qualitative inquiry may explore retrospective experiences and perceptions of the intervention.
Each arrangement has different strengths and limitations.
Interviews conducted during implementation may capture experiences close to the events being investigated, but participation in interviews could influence subsequent behavior.
Post-trial interviews may permit reflection on the overall intervention but can be affected by recall and knowledge of outcomes.
Researchers should select timing according to the information needed rather than assuming that qualitative collection must occur concurrently with quantitative outcome measurement.
How Should Qualitative Participants Be Selected?
Qualitative sampling should follow the supporting research question rather than automatically mirror the trial's statistical sampling strategy.
For example, if the purpose is to investigate implementation, researchers might purposively recruit participants from different classes, instructors, or delivery settings.
If the purpose is to understand intervention experiences, sampling may seek variation in engagement, accessibility, or reported experiences.
Palinkas and colleagues (2015) discuss purposeful sampling strategies that can be relevant to mixed-methods implementation research.
Researchers should consider whether the sample adequately represents the variation needed for the qualitative analysis.
A small group of enthusiastic participants may provide useful accounts but may not capture the experiences of those who disengaged or encountered substantial barriers.
Should Both Experimental Groups Be Interviewed?
Not necessarily. The answer depends on the qualitative purpose.
If the question concerns how participants experience a particular intervention, interviewing the intervention group may be appropriate.
If the question concerns differences in implementation conditions, expectations, or experiences across experimental conditions, including participants from both groups may be useful.
However, qualitative comparisons between groups should not automatically be interpreted as causal estimates.
Randomization supports causal inference for appropriately defined trial comparisons under relevant assumptions. A purposively selected interview subsample may not preserve the balance created by random allocation.
Researchers should distinguish the trial's experimental comparison from qualitative comparisons based on selected participants.
What Happens When Participants Are Selected According to Their Outcomes?
Researchers may wish to interview participants who improved substantially, showed little improvement, or experienced unexpected outcomes.
Such selection can support focused investigation of contrasting experiences.
However, selecting participants according to post-randomization outcomes introduces interpretive complications.
Outcome-defined groups may differ in baseline characteristics, adherence, motivation, or other factors. Randomization does not guarantee comparability between subgroups created after observing outcomes.
Consequently, interviews with high-performing and low-performing participants may generate hypotheses about possible processes but do not independently establish why the outcomes differed.
The selection strategy and its limitations should be reported explicitly.
How Can Qualitative Inquiry Investigate Implementation Fidelity?
Implementation fidelity concerns the extent to which an intervention was delivered or enacted in relation to its intended components.
Qualitative interviews and observations may investigate whether teachers followed planned procedures, how participants encountered the intervention, and why adaptations occurred.
For example, teachers may report modifying AI-generated feedback activities because of assessment requirements or students' digital access.
These accounts can help explain how implementation varied.
However, self-reported fidelity may differ from observed implementation. Researchers may need additional evidence, such as observation records, delivery logs, or appropriate quantitative indicators.
Qualitative accounts should be interpreted according to their evidentiary strengths rather than treated as complete verification of what occurred.
Can Qualitative Research Explain Why an Intervention Worked?
It can contribute evidence about plausible processes, but it does not automatically identify causal mechanisms.
Suppose an experimental trial estimates improved writing performance among students assigned to AI-generated feedback.
Interviews may suggest that students used feedback to identify weaknesses in argument structure or revise drafts more systematically.
These accounts may be consistent with a proposed mechanism, but they do not establish that the reported practices mediated the intervention effect.
Causal mediation claims require appropriate design, measurement, assumptions, and analytical evidence.
Skivington and colleagues (2021) emphasize the importance of program theory, context, and appropriate evaluation in complex intervention research.
Qualitative findings can refine program theory and identify possible mechanisms for subsequent investigation without being presented as definitive causal proof.
What If the Trial Finds No Clear Intervention Effect?
Qualitative evidence may still be informative.
For example, interviews might reveal inconsistent implementation, limited engagement, or substantial participant burden.
These findings may identify conditions relevant to interpreting the intervention and designing future evaluations.
However, they do not establish that the intervention would have been effective under different conditions.
A statistically imprecise treatment estimate remains imprecise even when participants describe positive experiences.
Researchers should distinguish evidence about implementation from evidence about effectiveness and avoid using qualitative accounts to rescue an unsupported experimental hypothesis.
How Can Qualitative Research Affect Trial Conduct?
Embedding qualitative inquiry may influence the trial itself, particularly when interviews prompt reflection or researchers share emerging findings with implementation staff.
For example, teachers who participate in interviews about their use of AI-generated feedback may reconsider their practices during the intervention period.
Such effects are not inevitable, but they should be considered when designing collection procedures.
Researchers should decide whether qualitative findings will be used for ongoing implementation improvement or reserved for later interpretation.
If findings are used to modify the intervention during the trial, the changes may affect the treatment being evaluated and should be documented.
Where maintaining intervention consistency is important, the research protocol should specify how qualitative feedback will be handled.
How Should Qualitative and Experimental Findings Be Integrated?
Integration should relate the qualitative findings to the experimental question without conflating their inferential purposes.
For example, a joint display may present quantitative intervention outcomes alongside qualitative evidence about implementation, participant experiences, and contextual conditions.
Guetterman, Fetters, and Creswell (2015) discuss joint displays as a means of supporting mixed-methods integration.
However, simply placing trial results beside interview quotations does not establish meaningful integration.
The researcher should explain how qualitative evidence informs interpretation of intervention delivery, feasibility, or participant experiences and identify what remains uncertain.
Where findings diverge, the disagreement should be investigated rather than concealed.
Does the Qualitative Component Need to Be Large?
There is no universal sample-size requirement for qualitative inquiry embedded within trials.
The appropriate sample depends on the qualitative question, sampling strategy, contextual variation, and analytical approach.
Malterud, Siersma, and Guassora (2016) discuss information power as one approach to considering qualitative sample adequacy.
A narrowly focused investigation of one implementation process may require different sampling from a study comparing experiences across multiple institutions and participant groups.
The qualitative component should be sufficiently developed to support its intended contribution, even when the experimental evaluation receives greater methodological priority.
When Is Embedded Qualitative Inquiry Unnecessary?
Qualitative inquiry may be unnecessary when the primary experimental question can be answered adequately without additional evidence about implementation, experiences, or context.
For example, a tightly controlled evaluation of a well-defined intervention may not require interviews if the supporting questions are already adequately addressed through existing evidence or appropriate trial measures.
Adding interviews solely to make the study appear more comprehensive may create unnecessary participant burden and analytical complexity.
Researchers should also consider whether a separate qualitative study would be more appropriate when the proposed inquiry has an independent purpose that is not primarily supportive of the trial.
The broader principles of an embedded mixed-methods design require a meaningful relationship between the primary and supporting components.
Watch Out
Do not use qualitative interviews as proof that an intervention caused its observed outcomes or would have succeeded under different implementation conditions. Qualitative evidence may illuminate experiences and plausible processes, but causal claims must remain consistent with the experimental design and the evidence supporting them.
