03 · What You Need to Know
When Nonrepresentative Samples Can Provide Strong Research Evidence
What Is a Nonrepresentative Sample?
A nonrepresentative sample is one that does not provide an adequate basis for representing a specified population for the characteristic or inference of interest.
However, representativeness is not always directly observable.
A sample may differ from the population on measured characteristics without necessarily producing biased estimates for every outcome.
Conversely, a sample may closely resemble the population on demographic variables while differing substantially on an unmeasured characteristic central to the research.
For example, a volunteer sample may match the university's distribution of academic disciplines while disproportionately including frequent generative AI users.
Researchers should therefore specify the population and inferential objective before labeling a sample representative or nonrepresentative.
Why Does Nonrepresentativeness Not Automatically Mean Invalidity?
Research validity is multidimensional.
A study may seek to estimate population characteristics, identify causal effects, examine mechanisms, understand experiences, or develop theoretical explanations.
These objectives require different evidence.
For example, a purposively selected group of teachers who have experienced AI-related assessment disputes may provide detailed evidence about institutional decision-making.
The sample need not statistically represent all teachers for the study to analyze those experiences rigorously.
However, the researcher should not use the interviews to estimate how frequently such disputes occur among all teachers.
The sample's adequacy depends on the conclusion being drawn.
Which Research Questions Can Nonrepresentative Samples Address?
| Research objective |
Potential contribution |
Important limitation |
| Qualitative experiences |
Detailed understanding of relevant perspectives and processes |
Does not directly estimate population frequencies |
| Theoretical development |
Identification and refinement of concepts or mechanisms |
Requires careful theoretical justification and contextual interpretation |
| Randomized experimental effects |
Causal comparison within an appropriately defined study population |
Broader population effects may differ |
| Methodological feasibility |
Evidence about procedures, usability, or recruitment processes |
Feasibility may vary across populations and settings |
| Instrument development |
Initial evidence about item functioning and measurement properties |
Further validation may be needed in intended populations |
| Prediction model development |
Learning predictive relationships in available data |
Performance and calibration may change in deployment populations |
| Population prevalence estimation |
Descriptive information about respondents |
Population estimates require additional selection assumptions or evidence |
The same sample may support some objectives well and others poorly. Researchers should distinguish the evidentiary contribution from the breadth of the claims.
Can a Nonrepresentative Sample Support Qualitative Research?
Yes. Many qualitative methodologies deliberately select participants with relevant experiences, roles, or characteristics.
For example, researchers examining institutional AI governance may interview administrators who participated in policy development.
These participants may represent only a small and unusual portion of university employees.
Nevertheless, their involvement makes them particularly relevant to understanding the policy development process.
Qualitative rigor depends on methodological coherence, appropriate evidence, reflexivity, and credible analysis rather than demographic proportionality alone.
However, researchers should explain why the selected cases are relevant and how their characteristics shape interpretation.
How Does Purposive Sampling Support Research Validity?
Purposive sampling selects participants or cases because they can contribute information relevant to the research question.
For example, a study investigating AI-assisted manuscript preparation may recruit researchers who have used generative AI during drafting, revision, or peer-review preparation.
Participants without such experience may be less able to describe the phenomenon firsthand.
Purposive selection can therefore strengthen the relevance of qualitative evidence.
However, the criteria should not selectively exclude perspectives that challenge the researcher's expectations.
Researchers should consider whether contrasting cases and variation in experience are necessary for a credible interpretation.
Can a Small, Focused Sample Be More Informative Than a Large Convenience Sample?
Yes, for certain research objectives.
For example, 20 experienced institutional AI policy developers may provide more relevant evidence about policy implementation than 2,000 randomly encountered individuals with little involvement in the process.
However, this does not mean small samples are generally superior.
Population prevalence estimation, rare-event analysis, and precise subgroup comparisons may require substantially larger samples and appropriate selection designs.
The relevant question is whether the sample contains sufficient information for the intended analysis.
Sample size and sampling method should therefore be justified together.
What Is Information Power?
Information power is a concept proposed by Malterud and colleagues to explain qualitative sample adequacy in relation to the information needed for a particular study.
A focused research question involving participants with highly relevant experiences may require fewer interviews than a broad study spanning many settings and experiences.
Information power considers the study aim, sample specificity, theoretical background, dialogue quality, and analytical strategy.
It does not provide a universal numerical threshold.
Researchers should use it as a framework for methodological justification rather than a formula for determining sample size.
Can Nonrepresentative Samples Support Causal Inference?
Yes, under appropriate designs and assumptions.
For example, a randomized trial may recruit eligible volunteers and randomly assign them to an AI tutoring intervention or comparison condition.
Random assignment can support a causal comparison within the study population by reducing systematic baseline confounding in expectation.
However, volunteers may differ from the broader population in motivation, prior achievement, or digital access.
The treatment effect estimated among participants may therefore differ from the effect in the population where the intervention will eventually be implemented.
Internal causal validity and population generalizability should be assessed separately.
Why Is Random Assignment Different From Representative Sampling?
Random assignment determines which intervention participants receive after enrollment.
Representative sampling concerns the relationship between the selected participants and the intended population.
For example, 150 volunteers may be randomly assigned to two instructional conditions.
The treatment allocation is randomized, but the participants were not necessarily selected randomly from all eligible students.
Researchers should not describe the sample as representative merely because the experimental groups were randomly assigned.
These procedures address different methodological requirements.
Can a Nonrepresentative Observational Sample Support Causal Claims?
Potentially, but causal inference requires more than an appropriate sample.
Researchers must evaluate confounding, selection mechanisms, exposure definition, measurement, and other identification assumptions.
For example, a study of voluntary AI tutoring use may compare users and nonusers within one institution.
Even if the sample is not representative nationally, a causal effect for the study population may be identifiable under suitable assumptions.
However, differences in motivation, baseline achievement, and access may confound the comparison.
Nonrepresentativeness does not automatically invalidate causal inference, but neither does it excuse inadequate confounding control.
When Does Selection Bias Become a Serious Concern?
Selection bias arises when selection or conditioning processes distort the relationship or quantity being estimated.
For example, selecting only students who complete an intervention may distort a treatment-effect estimate if completion is influenced by treatment and factors related to the outcome.
Similarly, a volunteer survey of AI adoption may overrepresent students who are particularly interested in AI.
However, these situations involve different inferential problems.
The first concerns a potentially biased causal comparison, while the second concerns population estimation from a selected respondent group.
Researchers should identify the relevant selection mechanism rather than treating every nonrepresentative sample as having the same bias.
What Is the Difference Between Selection Bias and Limited Generalizability?
Selection bias concerns distortion of an intended estimate or relationship arising from the selection process.
Limited generalizability concerns uncertainty about whether findings apply beyond the population or setting directly studied.
For example, a randomized trial restricted to students with reliable connectivity may estimate an internally valid effect among those participants.
However, the effect may not apply to students with intermittent access.
The limitation does not necessarily mean the trial estimate is biased for its defined study population.
Researchers should distinguish these issues because they require different methodological responses.
Can a Nonrepresentative Sample Estimate Population Prevalence?
It may be possible under additional assumptions and suitable statistical methods, but population prevalence cannot ordinarily be inferred from the respondent proportion alone.
For example, a volunteer survey may find that 80% of respondents use generative AI.
If frequent users are more likely to participate, the respondent proportion may overestimate population adoption.
Calibration, weighting, or model-based methods may address some selection differences when relevant information is available.
However, these methods cannot guarantee the elimination of bias from unmeasured selection factors.
Researchers should avoid presenting a volunteer proportion as a population prevalence estimate without appropriate justification.
Can Weighting Make a Nonrepresentative Sample Representative?
Weighting can sometimes improve population estimation by adjusting for measured differences between the sample and target population.
For example, a university survey may be calibrated to known distributions of academic year and discipline.
However, matching these distributions does not guarantee that respondents resemble nonrespondents in their AI use.
Weighting requires suitable population information, sufficient overlap, and defensible assumptions about selection and outcomes.
It cannot automatically recover information about groups entirely absent from the data.
Researchers should distinguish statistical adjustment from evidence that the original selection mechanism was probability-based.
What Is the Difference Between Sample Bias and Sample Size?
Sample size affects statistical precision and the amount of information available for analysis.
Selection bias concerns systematic distortion associated with how observations enter the study.
For example, a survey of 10,000 AI enthusiasts may produce a very precise estimate of AI use among those respondents.
It does not automatically estimate AI use among all university students.
Increasing the number of similarly selected respondents may reduce random uncertainty without addressing systematic selection.
A large sample should not be treated as a substitute for a suitable selection mechanism.
Can a Nonrepresentative Sample Be Useful for Prediction Models?
Yes, but the intended deployment population matters.
For example, a dropout prediction model may be developed using records from one university.
The model may identify useful predictive relationships in that dataset.
However, its performance may differ at institutions with different student populations, academic policies, or outcome definitions.
External validation can help assess whether discrimination, calibration, and other performance measures remain appropriate.
Researchers should avoid assuming that predictive performance in a selected development sample guarantees performance elsewhere.
When Can Nonrepresentative Sampling Support Instrument Development?
Instrument development may require participants who provide information across the range of the intended construct.
For example, an AI literacy assessment may initially be evaluated among students with varied levels of technology experience.
Deliberately including participants at the extremes of competence may help identify poorly functioning items.
However, subsequent validation should examine whether the instrument supports the intended interpretations in relevant populations.
A selected development sample does not automatically establish population norms or measurement equivalence.
What About Case Studies?
Case studies may investigate particular institutions, programs, events, or processes in depth.
For example, a researcher may examine how one university introduced a generative AI assessment policy.
The institution need not statistically represent all universities for the study to contribute evidence about implementation processes.
However, the case selection should be justified and contextual features described.
Researchers should distinguish conclusions about the case from broader theoretical propositions or claims about other institutions.
Analytical relevance does not automatically establish population prevalence.
Can Nonrepresentative Samples Support Theory Development?
Yes. Theory development may benefit from cases selected because they reveal mechanisms, contrasts, or conditions relevant to an emerging explanation.
For example, researchers studying AI governance may compare institutions with different policy approaches.
The cases may be deliberately selected to examine contrasting organizational processes rather than to reproduce the national distribution of universities.
However, theoretical claims should remain grounded in the evidence and methodological approach.
Researchers should consider alternative explanations and avoid treating selected cases as proof that a mechanism operates universally.
When Is Convenience Sampling Defensible?
Convenience sampling may be appropriate for certain feasibility studies, preliminary investigations, and focused research objectives.
For example, a researcher may evaluate whether a new digital assessment interface functions as intended among accessible users.
However, the sample should include participants capable of revealing the relevant usability or procedural problems.
If the technology is intended for students with diverse accessibility needs, testing only highly proficient users may leave important problems undetected.
Convenience should be acknowledged as a selection limitation rather than presented as evidence of population coverage.
When Is Purposive Sampling More Defensible Than Convenience Sampling?
Purposive sampling is particularly appropriate when the research requires participants with defined experiences, roles, or characteristics.
For example, investigating how research ethics committees evaluate generative AI protocols may require committee members with relevant review experience.
Recruiting whichever university employees are available would not necessarily provide comparable information.
Purposive selection should be justified through the research question and a transparent explanation of why selected participants are informative.
However, purposive sampling does not automatically eliminate researcher selection bias or guarantee conceptual completeness.
How Can Researchers Strengthen a Nonrepresentative Sample?
Begin by identifying the intended inference and selecting participants or cases relevant to it.
For qualitative research, consider variation in experiences, contrasting cases, and the adequacy of information for the analysis.
For experiments, prioritize valid treatment assignment, appropriate outcome measurement, and management of attrition.
For prediction, ensure relevant case mix and suitable external validation.
For population estimation using nonprobability data, evaluate available auxiliary information and the assumptions required for adjustment.
Improvement should target the actual inferential weakness rather than merely increasing sample size or demographic diversity.
How Should Researchers Define the Population of Inference?
The population of inference should reflect the people, cases, or settings to which the intended conclusion applies.
For example, a randomized intervention among students at one university may directly estimate an effect for the enrolled trial population.
Claims about all undergraduate students require additional assumptions concerning differences between participants and the broader population.
Similarly, a qualitative study may seek an explanation relevant to particular institutional contexts rather than a statistical estimate for all universities.
Researchers should define the intended scope before recruitment and revise claims transparently when the achieved sample provides narrower evidence.
How Does Nonrepresentativeness Affect External Validity?
External validity concerns whether findings can be applied beyond the study's immediate population or conditions.
Nonrepresentativeness may limit external validity when the sample differs from the target population in characteristics that influence the relevant outcome or treatment effect.
For example, an AI tutoring intervention may have different effects among novice and experienced users.
A trial including only experienced users may not directly establish the effect among novices.
However, differences in irrelevant characteristics need not prevent generalization.
Researchers should evaluate plausible effect modifiers and the assumptions needed to extend findings.
Can Findings Be Generalized Analytically Rather Than Statistically?
Yes. Analytical generalization concerns the relevance of findings to theoretical propositions, mechanisms, or conceptual explanations.
For example, a qualitative study of institutional AI governance may identify decision processes that inform analysis of other universities.
Such transfer requires consideration of context, mechanisms, and supporting evidence.
It does not establish how frequently those processes occur across all institutions.
Researchers should distinguish analytical transfer from statistical estimation and avoid using generalization as though it referred to one identical inferential procedure.
How Can Researchers Evaluate Whether Their Sample Is Adequate?
Ask whether the sample contains the information, variation, and comparisons required by the research question.
For a randomized trial, evaluate the treatment assignment mechanism, sample size, outcome measurement, and retention.
For qualitative research, assess participant relevance, information richness, and analytical sufficiency.
For prediction, examine case mix, outcome information, and validation performance.
For population estimation, assess coverage, selection probabilities, nonresponse, and assumptions supporting inference.
The same adequacy standard should not be imposed indiscriminately across different methodologies.
When Is a Nonrepresentative Sample Inadequate?
A nonrepresentative sample is inadequate when its selection mechanism or composition prevents the study from supporting its intended conclusion.
For example, a survey estimating university-wide AI adoption may be inadequate if it recruits only students enrolled in AI-focused courses and lacks a defensible adjustment strategy.
A qualitative study may also be inadequate if it excludes experiences essential to understanding the phenomenon.
A randomized trial may provide insufficient evidence for broader implementation when relevant target groups are absent.
Researchers should identify the specific inferential limitation rather than declaring the sample invalid solely because it is nonrepresentative.
How Should Nonrepresentative Sampling Be Reported?
Methods sections should describe the target population, eligibility criteria, sampling approach, recruitment channels, and achieved sample.
Explain why the selection strategy was appropriate for the research question.
Where relevant, report important differences between participants and the intended broader population.
Describe limitations involving coverage, self-selection, unmeasured confounding, or transferability as appropriate to the design.
Conclusions should distinguish what the data establish from what would require additional population evidence.
Researchers should avoid presenting nonrepresentative samples as universally representative or dismissing their legitimate contributions.
