03 · What You Need to Know
How Sampling Priorities Change Across Research Objectives
What Does It Mean to Optimize a Sampling Strategy?
Sampling optimization means designing the selection of participants or other units to obtain information appropriate for a specified research objective within practical and ethical constraints.
It does not necessarily mean maximizing sample size, achieving equal subgroup proportions, or minimizing recruitment costs.
For a survey, optimization may concern the precision and bias of a population estimate. For an experiment, it may involve recruiting eligible participants who can contribute to a valid treatment comparison. For qualitative inquiry, it may concern selecting cases that clarify a phenomenon or challenge an emerging explanation.
Researchers should identify the criterion against which a sampling strategy is being evaluated before describing it as optimal.
Why Must the Research Question Come Before the Sampling Method?
The research question establishes the information needed and the scope of the intended conclusion.
For example, asking what proportion of students use generative AI requires evidence about the distribution of AI use in a defined student population.
Asking why students trust AI-generated feedback requires information about experiences, interpretations, and relevant contexts.
Both questions concern AI use, but their sampling requirements differ.
Choosing the method first risks designing a study around accessible participants rather than around the evidence needed to answer the question.
How Do the Four Sampling Objectives Compare?
| Objective |
Primary sampling priority |
Potentially suitable approach |
Principal concern |
| Population estimation |
Coverage, known selection probabilities, precision |
Probability sampling, including stratified or multistage designs |
Coverage error, nonresponse, sampling uncertainty |
| Causal inference |
Identifiable treatment or exposure contrast and relevant population |
Eligibility-based recruitment with appropriate experimental or observational design |
Confounding, selection, positivity, external validity |
| Prediction |
Relevant case mix, outcome information, transportability, validation |
Population-aligned cohorts, enriched development samples where justified |
Overfitting, calibration, dataset shift |
| Conceptual depth |
Information-rich cases, variation, theoretical relevance |
Purposive, maximum-variation, or theoretically guided sampling |
Insufficient conceptual coverage or superficial evidence |
These categories are not mutually exclusive. A mixed methods study may combine probability sampling for a survey with purposive sampling for interviews, while a causal study may also require population-representative recruitment for a policy-relevant treatment effect.
What Should Sampling Prioritize for Population Estimation?
Population estimation seeks quantities such as proportions, means, totals, or distributions for a defined population.
For example, a researcher may estimate the proportion of undergraduate students who use generative AI weekly.
The sampling strategy should provide appropriate coverage of the intended population and a defensible basis for estimating the population quantity.
Probability sampling is often preferable when a suitable frame and feasible selection procedure exist because the design establishes known inclusion probabilities.
However, random selection does not automatically resolve nonresponse, measurement error, or incomplete coverage.
The appropriate objective is not simply a sample that looks demographically balanced, but an estimator with defensible properties for the target quantity.
How Does Statistical Precision Influence Population Sampling?
For a population estimate, researchers may seek to minimize sampling variance for a given budget or achieve a desired precision with available resources.
For example, stratification may improve precision when the stratification variable is strongly associated with the outcome.
Cluster sampling may reduce fieldwork costs but can increase variance when observations within clusters are correlated.
Unequal selection probabilities may improve information about small subgroups while requiring appropriate weighting.
Optimization therefore involves trade-offs between cost, coverage, precision, and the estimates that matter most.
Does the Most Representative Sample Always Produce the Most Precise Estimate?
No. Representativeness and statistical precision are related but distinct.
A disproportionate stratified probability sample may deliberately oversample a small subgroup to improve subgroup precision.
With suitable weights and variance estimation, the design may still support population-wide inference.
Conversely, a demographically balanced volunteer sample may provide a precise respondent estimate while remaining vulnerable to selection bias.
Researchers should distinguish sample composition, design-based precision, and systematic error.
What Should Sampling Prioritize for Causal Inference?
Causal inference concerns the effect of an exposure or intervention under a specified comparison.
For example, a researcher may seek the average effect of AI tutoring compared with conventional academic support.
The sampling strategy must establish a population for which the treatment contrast is meaningful, but causal identification also requires appropriate treatment assignment or assumptions about observational exposure selection.
A probability sample cannot, by itself, eliminate confounding.
Similarly, random assignment among volunteers may support an internally valid causal comparison without making the volunteers representative of a broader population.
Researchers should distinguish selection into the study from assignment to treatment.
What Is the Difference Between Sampling and Random Assignment?
Sampling determines which units enter the study. Random assignment determines which treatment or intervention enrolled units receive.
Suppose 200 eligible students volunteer for a trial and are randomly assigned to AI tutoring or conventional instruction.
The assignment mechanism may support causal identification within the enrolled population, subject to the design and relevant assumptions.
However, the sample remains volunteer-based relative to the broader university population.
Random assignment does not automatically establish population representativeness, and random sampling does not automatically establish causation.
How Does the Estimand Affect Causal Sampling?
An estimand specifies the causal quantity of interest, including the treatment contrast, outcome, population, and other relevant attributes.
For example, the average effect among students who volunteer for an intervention is not necessarily identical to the average effect among all students who would receive it under institutional implementation.
If prior achievement or digital access modifies treatment effects, differences between these populations may matter.
Researchers should specify whether the target is an effect among enrolled participants, eligible patients, a policy population, or another defined group.
Sampling decisions should be evaluated against that population component.
What Is Positivity, and Why Does It Matter?
Positivity concerns whether the treatment or exposure alternatives relevant to a causal estimand are possible within the covariate patterns being compared.
For example, if students with certain characteristics never receive AI tutoring, estimating its observational causal effect for those students may require extrapolation beyond the available comparisons.
Restricting the target to a population with adequate overlap may sometimes be defensible, but doing so changes the population-specific estimand.
Researchers should not broaden recruitment without considering whether the intended treatment contrast can be identified for newly included groups.
Can Deliberately Oversampling Certain Groups Improve Causal Research?
Yes, when subgroup effects or effect heterogeneity are important objectives.
For example, researchers may deliberately recruit more students with limited digital literacy to estimate intervention effects within that subgroup with greater precision.
However, the resulting sample proportions may differ from those in the intended population.
Estimating a population average effect may therefore require appropriate information about population composition and additional assumptions.
Oversampling can improve one inferential objective while complicating another.
What Should Sampling Prioritize for Prediction?
Prediction studies aim to estimate outcomes for future individuals or observations using available predictors.
For example, a model may predict which students are at risk of withdrawing from an online course.
The sampling strategy should provide sufficient information about the predictors, outcomes, and case mix relevant to the intended application.
Unlike a prevalence survey, the immediate objective may not be to estimate a population proportion.
However, the relationship between the development sample and deployment population remains important because predictive performance can change when population characteristics or outcome processes differ.
Why Does Case Mix Matter for Prediction?
Case mix refers to the distribution of relevant participant characteristics and outcome risks.
For example, a dropout prediction model developed only among academically high-performing students may have limited information about students with greater academic difficulties.
The model may perform differently when applied to a broader student population.
Differences in case mix can influence discrimination, calibration, and other measures of predictive performance.
Researchers should consider whether the development and validation data adequately represent the predictor and outcome patterns encountered during deployment.
Should Prediction Studies Oversample Rare Outcomes?
Sometimes. When an outcome is rare, an enriched development dataset may provide additional information about cases that would otherwise be scarce.
For example, researchers developing a model for an uncommon academic outcome may deliberately include more students experiencing that outcome.
However, altered outcome prevalence can affect probability calibration and the interpretation of model performance.
Appropriate analysis and validation are necessary, and oversampling cannot create information about predictor patterns absent from the source data.
Researchers should distinguish strategies that improve model development from those that establish reliable performance in the intended deployment population.
Why Is External Validation Important for Prediction Sampling?
A prediction model may perform well in its development dataset but less well in another institution, cohort, or time period.
External validation evaluates performance using data separate from the original development setting or process.
For example, a model developed using one university's online learning records may be evaluated using records from another institution.
Researchers should examine differences in predictor measurement, outcome definitions, student characteristics, and institutional conditions.
Validation data should be selected to answer a clear question about where and for whom the model is expected to work.
What Should Sampling Prioritize for Conceptual Depth?
Qualitative inquiry often seeks to understand experiences, meanings, processes, or social relationships.
For example, a study may investigate how researchers negotiate authorship responsibility when using generative AI.
Participants with relevant firsthand experiences may provide more useful evidence than a demographically proportional sample containing many people unfamiliar with the phenomenon.
Purposive selection may therefore emphasize experiential relevance, contrasting perspectives, or theoretically important cases.
The objective is not to maximize the number of participants but to obtain evidence capable of supporting a credible and sufficiently developed interpretation.
How Does Information Power Guide Qualitative Sampling?
Malterud and colleagues proposed information power as a framework for considering sample adequacy in qualitative interview studies.
The amount of information needed depends partly on the specificity of the research aim, relevance of the sample, theoretical background, dialogue quality, and analytical strategy.
A focused study involving participants with highly relevant experiences may require fewer interviews than a broad study spanning many settings.
However, information power does not prescribe a universal sample size.
Researchers should use it to justify the relationship between the sample and the intended analysis rather than as a numerical recruitment formula.
What Is Maximum-Variation Sampling?
Maximum-variation sampling deliberately selects cases that differ across characteristics relevant to the phenomenon.
For example, a qualitative study of institutional AI policies may include universities with different governance arrangements, resources, and implementation histories.
The purpose is to investigate how experiences or processes vary across contexts.
However, selecting contrasting cases does not establish their frequency in the broader population.
Maximum variation supports conceptual comparison, not automatic statistical representativeness.
When Is Theoretical Sampling Appropriate?
In methodologies such as grounded theory, theoretical sampling may involve selecting additional cases in response to emerging analytical categories.
For example, preliminary interviews may suggest that institutional authority shapes how researchers interpret AI authorship rules.
The researcher may then seek participants working under contrasting governance arrangements to examine that emerging explanation.
This approach differs from selecting a fixed demographic quota before analysis.
Its appropriateness depends on the qualitative methodology and the transparency of the analytical process.
Can One Study Optimize Sampling for Multiple Objectives?
Yes, although competing objectives may require explicit compromises.
For example, a mixed methods study may use a probability survey to estimate AI adoption and purposively select survey respondents for follow-up interviews.
The survey sample supports population estimation, while the interview subsample supports detailed exploration of selected experiences.
The two components need not use identical selection criteria.
However, the researcher should explain how the sampling designs relate and how the findings will be integrated.
What Happens When Sampling Objectives Conflict?
Sampling objectives can compete for limited resources.
For example, proportional allocation across university programs may support overall population estimation, while oversampling a small program may improve subgroup precision.
Likewise, recruiting a narrowly defined experimental population may simplify a causal comparison but limit direct applicability to broader users.
Researchers should identify which objective is primary and which are secondary.
Design compromises should be documented rather than presented as though one strategy simultaneously optimizes every possible inference.
How Does the Sampling Frame Constrain Optimization?
A sampling frame determines which units can be reached through a particular selection mechanism.
For example, an enrollment register may support probability selection of current students but omit individuals who recently withdrew.
If the research concerns dropout experiences, excluding withdrawn students from the frame may be particularly problematic.
Researchers should evaluate whether the sampling frame adequately covers the target population before optimizing allocation or sample size.
A highly efficient design applied to the wrong population cannot automatically support the intended inference.
Does a Larger Sample Improve Every Research Objective?
No. Larger samples can improve precision or provide additional information, but the benefits depend on the design.
A large volunteer survey may remain affected by systematic selection.
A large observational dataset may still contain unmeasured confounding.
A prediction model may still perform poorly in another population, while a qualitative study may accumulate repetitive interviews without improving conceptual understanding.
Researchers should distinguish the quantity of data from the relevance and validity of the information collected.
How Should Sample Size Be Planned for Different Objectives?
Population surveys may use precision-based calculations that incorporate expected variability and design effects.
Causal studies may require calculations based on the treatment contrast, expected effect, outcome variability, and allocation design.
Prediction studies require attention to the number of outcomes, model complexity, anticipated performance, and risks of overfitting.
Qualitative studies require a methodology-specific justification of analytical adequacy rather than automatic application of statistical power formulas.
The appropriate sample size follows the inferential objective and analytical plan.
How Can Researchers Evaluate a Proposed Sampling Strategy?
Begin by writing down the intended claim and the population or cases to which it applies.
Identify the selection mechanism and the assumptions needed to support that claim.
Then consider what would make the estimate or interpretation unreliable: incomplete population coverage, invalid causal comparisons, inadequate prediction case mix, or insufficient qualitative variation.
Finally, evaluate whether an alternative sampling approach would address those weaknesses within available resources.
The objective is to identify the most consequential inferential risk, not to pursue representativeness, diversity, or sample size as isolated ideals.
How Should Sampling Optimization Be Reported?
Methods sections should identify the primary research objective, target population, sampling units, selection procedures, and rationale for the chosen strategy.
Where relevant, report stratification, clustering, unequal selection probabilities, weighting, recruitment procedures, and nonresponse.
For causal research, distinguish participant selection from treatment assignment and identify the population component of the estimand.
For prediction, describe the development and validation populations and relevant sampling decisions.
For qualitative inquiry, explain why selected participants or cases were appropriate for the intended conceptual analysis.
Readers should be able to understand what the sampling strategy was designed to accomplish and what limitations remain.
