03 · What You Need to Know
When Qualitative Diversity Becomes Analytically Unmanageableable
What Does Heterogeneity Mean in Qualitative Sampling?
Heterogeneity refers to differences among selected cases, participants, settings, experiences, or other units of analysis.
These differences may involve professional roles, institutional contexts, disciplinary backgrounds, stages of experience, or relationships to the phenomenon.
For example, a study of AI-supported assessment may include instructors from several disciplines and institutions with contrasting AI policies.
Such variation may be useful when the research question concerns how contextual conditions shape implementation.
However, heterogeneity is relative to the characteristics considered important. Participants may differ demographically while sharing the same defining experience, or appear demographically similar while encountering substantially different institutional conditions.
The methodological significance of heterogeneity depends on its relationship to the research question.
What Makes a Sample Excessively Heterogeneous?
A sample becomes excessively heterogeneous when its variation introduces analytical demands that the research design cannot adequately support.
For example, a study involving 15 participants from ten different institutional contexts may provide insufficient information about the conditions operating within each institution.
If the researcher intends to explain institutional differences, limited contextual evidence may undermine the interpretation.
However, the same sample might be adequate for a narrower question concerning a shared experience that crosses those settings.
There is no universal numerical threshold at which a sample becomes too heterogeneous.
The concern arises from a mismatch between the range of cases, the intended analytical claims, and the evidence available.
Is Heterogeneity the Same as Maximum Variation Sampling?
No. Maximum variation sampling deliberately selects cases differing along dimensions relevant to the research question.
Heterogeneity describes the presence of differences, whether those differences were deliberately selected or arose incidentally.
Purposeful maximum variation
Includes analytically relevant contrasts selected to illuminate differences or recurring patterns across cases.
Excessive heterogeneity
Introduces more or different forms of variation than the study can examine adequately for its intended purpose.
A maximum variation design may become excessively heterogeneous if too many dimensions are introduced without sufficient data or analytical focus.
The distinction between maximum variation and homogeneous sampling is therefore important when deciding which differences should guide recruitment.
Why Can Excessive Heterogeneity Reduce Analytical Depth?
Different participant groups and contexts may require different questions, contextual information, and interpretive frameworks.
For example, university instructors may discuss assessment design, while technical personnel focus on system reliability and administrators describe institutional governance.
These perspectives may all be relevant to a broad implementation study.
However, if the research question concerns instructors' lived experiences of AI-generated feedback, including many participants who have never delivered such feedback may dilute the analytical focus.
The researcher may spend substantial effort describing differences in participant roles rather than developing a detailed interpretation of the intended phenomenon.
Excessive heterogeneity can therefore create breadth without adequate explanatory depth.
Does a Heterogeneous Sample Always Require More Participants?
Not necessarily, but greater analytical variation may increase the evidence required to examine relevant contrasts.
Suppose a study compares experiences across four institutional sectors. If the analysis intends to characterize each sector, it needs sufficient evidence within those contexts.
Adding more sectors without additional data may weaken the basis for comparison.
However, sample size alone cannot determine adequacy.
Repeated interviews, observations, documents, and contextual records may provide additional depth without simply increasing the number of individual participants.
The relevant question is whether the available evidence supports the comparisons and interpretations the study intends to make.
How Does Information Power Relate to Heterogeneity?
Malterud, Siersma, and Guassora's information power framework emphasizes that sample adequacy depends on the study aim, sample specificity, theoretical background, quality of dialogue, and analytical strategy.
A sample involving highly specific participants and a narrowly focused question may provide substantial relevant information.
A broader study involving multiple distinct contexts may require more extensive evidence to support its intended interpretation.
However, information power does not provide a mechanical rule for determining the maximum acceptable heterogeneity.
Researchers should evaluate whether the selected cases provide sufficient information for the analytical scope.
The framework encourages attention to the relationship between sample specificity and the purpose of the inquiry rather than treating diversity as inherently beneficial.
How Can Too Many Sampling Dimensions Create Problems?
Sampling dimensions are characteristics deliberately used to structure case selection.
Suppose a researcher wants variation across five disciplines, four institutional types, three career stages, and two AI implementation conditions.
A complete cross-classification would create 120 possible combinations.
Most qualitative studies would not seek to populate every combination, but the example illustrates how rapidly the potential analytical structure can expand.
If the research attempts to compare many combinations without sufficient evidence, the analysis may become fragmented.
Researchers should therefore prioritize dimensions that are theoretically or substantively relevant and avoid unnecessary cross-classification.
What Happens When Each Subgroup Contains Very Few Participants?
Small subgroup counts are not automatically problematic in qualitative research.
A single critical case may provide important evidence for a particular analytical question.
However, difficulties arise when the researcher makes broad claims about a subgroup based on limited and insufficiently contextualized evidence.
For example, interviewing one instructor from each of eight disciplines may not support detailed conclusions about how those disciplines differ in AI assessment practices.
The participants may provide useful individual accounts, but their experiences should not automatically be treated as characteristic of their disciplines.
Researchers should distinguish the analytical value of individual cases from claims about group-level patterns.
Can Heterogeneity Make Interviews Difficult to Compare?
Yes, particularly when participants have substantially different relationships to the phenomenon.
For example, an administrator discussing institutional policy development and a student discussing personal feedback experiences may answer different analytical questions.
Comparing their accounts as though they describe the same experience can obscure important distinctions.
However, different perspectives may be essential in a multi-stakeholder case study.
The researcher should establish whether the intended comparison concerns shared experiences, contrasting roles, or relationships between institutional processes.
Comparability should be defined analytically rather than assumed from the fact that all participants discuss the same general topic.
How Does the Research Question Determine Acceptable Heterogeneity?
A broad question may legitimately require more contextual variation than a narrowly defined question.
For example, investigating how different educational institutions govern generative AI may require administrators, instructors, students, and relevant documents across several settings.
By contrast, investigating how first-year instructors experience AI-generated feedback during their initial semester may require a more focused participant group.
The same sample can therefore be appropriately heterogeneous for one research question and excessively heterogeneous for another.
Researchers should define the analytical boundaries before recruitment and revisit them when emerging evidence reveals a mismatch.
Can Heterogeneity Be Useful for Identifying Common Patterns?
Yes. Deliberate variation can help researchers examine whether particular experiences or processes appear across contrasting contexts.
For example, instructors from different institutional sectors may all describe uncertainty about accountability for AI-generated feedback.
This recurrence may suggest that accountability concerns are relevant across the selected contexts.
However, recurrence among purposively selected cases does not establish the prevalence of that concern in the wider population.
Researchers should also examine whether apparently similar themes have different meanings in different contexts.
A common label does not necessarily indicate an identical underlying process.
When Should Researchers Narrow the Sample?
Narrowing may be appropriate when the selected cases no longer align with the research question or when contextual variation prevents meaningful analysis.
For example, a study initially including teachers, students, administrators, and technical personnel may discover that its central analytical contribution concerns instructors' decisions about AI feedback verification.
The researcher may focus subsequent recruitment on instructors with direct verification experience.
However, narrowing should be analytically justified rather than used to exclude inconvenient or contradictory cases.
Researchers should document changes to eligibility criteria, sampling priorities, and analytical scope.
Where ethics approval specifies recruitment conditions, relevant amendments may be required.
Can Researchers Preserve Diversity While Improving Focus?
Yes. One approach is to establish a shared eligibility criterion while deliberately varying a limited number of important characteristics.
For example, all participants might be instructors who have implemented AI-generated feedback, while variation is sought across institutional policy environments.
This creates a common relationship to the phenomenon without eliminating contextual differences.
Another approach is to select contrasting institutional cases and conduct sufficiently detailed investigation within each.
Researchers should distinguish the level at which variation is introduced from the level at which depth is required.
The broader balance between qualitative sampling diversity and depth should guide these decisions.
How Does Excessive Heterogeneity Affect Saturation?
Heterogeneity can complicate claims of saturation because different contexts or participant groups may contribute different issues and meanings.
For example, a researcher may observe recurring codes among university instructors while continuing to encounter substantially different experiences among school teachers.
Claiming saturation across the entire sample may conceal important differences in analytical coverage.
Hennink, Kaiser, and Marconi distinguish code saturation from meaning saturation, highlighting that identifying recurring issues does not necessarily establish comprehensive understanding.
Researchers should evaluate adequacy according to the methodological approach and the specific claims being developed.
There is no universal requirement that every subgroup reach a separate numerical saturation threshold.
Can Excessive Heterogeneity Affect Transferability?
A heterogeneous sample may provide information about several contexts, but it does not automatically make findings transferable to a wider range of settings.
Transferability requires sufficient contextual description and analytical reasoning to evaluate whether findings may be relevant elsewhere.
If the study provides only superficial accounts of many settings, readers may lack the information needed to judge contextual similarity.
A narrower but well-contextualized study may sometimes provide a stronger basis for evaluating transferability than a broadly recruited study with limited contextual detail.
Researchers should avoid treating geographical or demographic breadth as proof of external applicability.
What If Heterogeneity Becomes Apparent Only During Analysis?
Researchers may discover during analysis that participants interpret the central phenomenon in substantially different ways.
For example, some participants may discuss AI feedback as a teaching tool, while others discuss it as an institutional monitoring system.
The researcher should examine whether these differences are analytically meaningful or indicate that the original research question was insufficiently focused.
Possible responses include refining the research question, developing clearly bounded analytical subgroups, collecting additional evidence, or limiting conclusions.
However, researchers should not retroactively redefine the sample merely to remove inconvenient findings.
Any changes should be transparent and consistent with the methodological approach.
How Should Excessive Heterogeneity Be Reported?
The methodology should describe the characteristics used to select participants and explain why variation was considered relevant.
If the achieved sample contains more contextual diversity than the analysis can adequately examine, the limitation should be acknowledged.
Researchers should distinguish findings supported across multiple cases from interpretations based on particular contexts or individual accounts.
Where recruitment or analytical boundaries changed, the reasons should be documented.
Reporting frameworks such as COREQ and the Standards for Reporting Qualitative Research provide useful guidance on describing participant selection and methodological limitations.
The strongest account explains how the sampling configuration affected the evidence available for interpretation.
