03 · What You Need to Know
The Factors That Determine How Many Observations a Study Requires
Why Must Sample Size Begin with the Research Question?
Sample size is a design decision rather than a number chosen independently of the research objective.
For example, estimating the percentage of students who use generative AI weekly requires information about the precision desired for a population proportion.
Comparing learning outcomes between students receiving AI-generated feedback and those receiving conventional feedback requires assumptions about the expected difference and outcome variability.
A qualitative investigation of how instructors interpret AI assessment policies requires a different rationale concerning the relevance and adequacy of selected cases.
The same participant count may therefore be adequate for one objective and inadequate for another.
Researchers should identify the primary question and intended analysis before selecting a recruitment target.
How Does the Type of Study Affect Sample Size?
Different research designs require different kinds of evidence.
Descriptive studies may prioritize precision when estimating means, proportions, or other population quantities.
Comparative and experimental studies may prioritize statistical power for detecting a substantively meaningful effect.
Predictive studies may require enough outcome information to develop and evaluate models without excessive overfitting.
Qualitative studies may prioritize information-rich cases and sufficient analytical depth.
Feasibility studies may focus on recruitment, retention, or procedural performance rather than confirmatory hypothesis testing.
The design determines which sample size considerations are most important.
How Does the Primary Outcome Affect Sample Size?
The outcome being measured strongly influences sample size requirements.
Continuous outcomes, such as assessment scores, require assumptions about variability and the intended mean difference or precision.
Binary outcomes, such as whether a student uses AI weekly, require assumptions about event prevalence.
Time-to-event outcomes may depend on the number of observed events, follow-up duration, and censoring.
Count outcomes may require assumptions about event rates and dispersion.
Researchers should therefore identify the primary outcome before choosing a calculation method.
Different outcomes within the same study may imply different sample size requirements.
How Does Effect Size Influence the Required Sample?
For many hypothesis-testing designs, smaller effects require larger samples to detect with a specified level of statistical power.
For example, detecting a small improvement in examination performance from an AI feedback intervention may require more participants than detecting a large improvement, assuming other conditions remain constant.
However, the expected effect should not be selected merely to produce a feasible sample size.
Researchers should distinguish the effect they anticipate from the smallest effect that would be substantively important.
The role of effect size in sample size calculation is particularly important when the study aims to test a specific intervention or association.
How Does Outcome Variability Affect Sample Size?
Greater variability generally makes it more difficult to distinguish a specified mean difference from random variation.
For example, student assessment scores may vary widely because of differences in prior knowledge, motivation, and educational context.
If the standard deviation is large relative to the difference the study aims to detect, more observations may be required.
For a simple comparison of two independent means with equal group sizes, the required sample per group is approximately proportional to the outcome variance when other planning assumptions remain fixed.
Consequently, doubling the assumed standard deviation can approximately quadruple the required sample size in that setting.
However, the relationship depends on the statistical model and design.
See how outcome variability affects sample size for a more detailed explanation.
How Does Desired Precision Affect Sample Size?
Precision concerns how narrowly a study aims to estimate an unknown quantity.
For example, a researcher estimating the prevalence of weekly AI use may consider a margin of error of five percentage points adequate for the intended purpose.
Another study may require a narrower margin because its findings will inform a consequential institutional decision.
Under common simple random sampling approximations, sample size is inversely proportional to the square of the desired margin of error.
Reducing the target margin of error from ten percentage points to five percentage points therefore requires approximately four times as many observations when other assumptions remain unchanged.
However, conventional probability-based precision statements depend on the sampling design and estimation assumptions.
How Does Statistical Power Affect Sample Size?
Statistical power is the probability that a specified testing procedure rejects the null hypothesis when a particular alternative is true, under the assumptions used for planning.
For example, a study designed for 90% power will generally require more participants than the same study designed for 80% power, assuming the effect size, significance level, and other inputs remain unchanged.
Power is not the probability that the research hypothesis is true.
It is also not a guarantee that the study will detect the effect in the realized dataset.
The importance of statistical power in study design lies in evaluating the operating characteristics of the planned test before data collection.
How Does the Significance Level Affect Sample Size?
The significance level, commonly denoted by alpha, specifies the test's nominal probability of rejecting the null hypothesis when it is true under the assumed statistical model.
A more stringent significance threshold generally requires a larger sample to maintain the same power for a specified effect.
For example, a test using alpha = 0.01 will ordinarily require more observations than an otherwise identical test using alpha = 0.05.
The appropriate threshold should follow the research context, consequences of false-positive findings, and any relevant multiplicity considerations.
Researchers should not adjust alpha merely to make an infeasible sample appear adequate.
See how significance level affects sample size.
Do One-Sided and Two-Sided Tests Require Different Samples?
They can. Under otherwise identical assumptions, a one-sided test may require fewer observations than a two-sided test for an effect in the prespecified direction.
However, choosing a one-sided test solely to reduce sample size is not defensible.
A one-sided test should be justified by the scientific question and the consequences of effects in the opposite direction.
For example, a researcher evaluating whether an AI intervention improves learning should consider whether a deterioration in learning outcomes would also be scientifically important.
The distinction between one-sided and two-sided tests affects planning and interpretation.
How Does Outcome Prevalence Affect Sample Size?
For a population proportion, the variance term p(1 − p) is largest when p = 0.50.
This is why 50% is commonly used as a conservative planning assumption for a simple proportion estimate when prevalence is unknown.
However, rare outcomes can create additional difficulties for regression, subgroup analysis, and comparisons because relatively few events may be observed.
For example, a study examining an uncommon adverse response to AI-generated feedback may require a large sample to obtain sufficient outcome events.
Thus, the relationship between prevalence and sample size depends on whether the objective is estimating a proportion, detecting a difference, or fitting a statistical model.
See how outcome prevalence affects sample size.
Does Population Size Determine Sample Size?
Population size can matter, but it is not always the dominant determinant.
For estimating a proportion under simple random sampling, the initial sample size requirement may be similar for large populations when the desired precision and assumed prevalence are the same.
For a relatively small finite population sampled without replacement, a finite population correction can reduce the required number of observations.
However, a population of 100,000 does not automatically require ten times the sample size of a population of 10,000.
Researchers should avoid selecting sample size solely as a fixed percentage of the population without a design-specific justification.
How Does the Number of Groups Affect Sample Size?
Studies comparing several groups may require different sample sizes from studies involving one or two groups.
For example, comparing AI feedback outcomes across four academic disciplines introduces different analytical requirements from estimating one overall mean.
The required number depends on the primary comparison, planned test, expected differences, variability, allocation, and multiplicity considerations.
More groups do not always imply a simple proportional increase in total sample size.
Researchers should specify which contrasts are primary rather than assume that every possible comparison requires equal precision or power.
See how the number of groups affects sample size.
How Does Unequal Allocation Affect Sample Size?
In many two-group comparisons, equal allocation is statistically efficient when observations have similar costs and variances.
Unequal allocation may require a larger total sample to achieve the same precision or power under those assumptions.
However, unequal allocation may be justified by recruitment feasibility, intervention costs, ethical considerations, or different outcome variances.
For example, a study may recruit more students from the control condition because the intervention is expensive to deliver.
The sample size calculation should reflect the intended allocation rather than assume equal group sizes.
See how unequal allocation ratios affect sample size.
How Do Repeated Measures Affect Sample Size?
Repeated-measures designs collect multiple observations from the same participant.
For example, a study may measure student writing performance before an AI intervention, immediately afterward, and at follow-up.
Repeated observations may improve efficiency when within-person measurements are correlated in ways that help estimate the target effect.
However, repeated measurements are not equivalent to independent participants.
The required sample depends on the number and timing of measurements, within-person correlation, outcome variability, missing data, and planned statistical model.
See how repeated measures affect sample size requirements.
How Does Clustering Affect Sample Size?
Clustering occurs when observations are grouped within shared units, such as students within classes or teachers within schools.
Participants in the same cluster may have correlated outcomes because they share instructional conditions, institutional resources, or other contextual influences.
As a result, a sample of 400 students drawn from a few classes may provide less independent information than 400 students selected under an appropriate independent sampling design.
The required sample depends on cluster sizes, intraclass correlation, number of clusters, and the intended analysis.
For cluster-randomized studies, the number of clusters may be particularly consequential.
The design effect provides one way to summarize changes in variance associated with a sampling design, although its form and applicability depend on the setting.
How Does the Sampling Method Affect Sample Size?
The sampling method influences the relationship between sample size and the information obtained.
Simple random sampling, stratified sampling, cluster sampling, and multistage sampling may produce different precision for the same nominal number of observations.
Stratification can improve precision when relevant strata are appropriately defined and analyzed.
Clustering may reduce effective information when observations within clusters are positively correlated.
Unequal selection probabilities may require weighting and influence estimator variance.
Non-probability sampling introduces additional selection assumptions that cannot be resolved merely by increasing the number of respondents.
The sample size calculation and subsequent analysis should therefore reflect the actual sampling design.
How Does Attrition Affect the Required Recruitment Target?
Attrition reduces the number of participants providing usable data at later study stages.
For example, a longitudinal study may require 200 participants with complete follow-up data but anticipate that some participants will withdraw or become unavailable.
Under a simple planning assumption of 20% attrition, the initial recruitment target would be 200 / 0.80 = 250 participants.
However, attrition can also introduce bias if withdrawal is related to outcomes or treatment conditions.
Increasing initial recruitment may preserve sample size but does not automatically correct informative attrition.
See adjusting sample size for dropout and attrition.
How Does Nonresponse Affect Sample Size?
Nonresponse affects the number of completed observations obtained from an initial set of invitations.
Suppose a survey requires 300 usable responses and anticipates a 60% usable-response rate among eligible invitees.
The corresponding invitation target is 300 / 0.60 = 500.
This calculation addresses expected recruitment yield, not the possibility of systematic differences between respondents and nonrespondents.
Researchers should distinguish the required completed sample from the number of invitations needed to obtain it.
The more detailed planning considerations are addressed in adjusting sample size for nonresponse.
How Does Analytical Complexity Affect Sample Size?
Complex statistical analyses may require more information than simple descriptive summaries.
For example, a regression model with many predictors, interactions, or nonlinear terms may require substantial data to estimate parameters reliably.
Prediction models also require attention to overfitting, outcome events, calibration, and anticipated predictive performance.
Universal rules such as a fixed number of participants per predictor may be inadequate because requirements depend on the model and outcome.
Researchers should specify the intended analysis before determining sample size.
Where conventional formulas are insufficient, simulation or specialized planning methods may be appropriate.
How Do Multiple Outcomes Affect Sample Size?
Studies often measure several outcomes, but these outcomes may imply different sample size requirements.
For example, an AI learning intervention may examine writing performance, student confidence, time spent completing assignments, and academic integrity concerns.
A sample adequate for detecting a meaningful difference in writing scores may be insufficient for a rare integrity-related outcome.
Researchers should identify the primary outcome or clearly specify a multiple-primary-outcome strategy.
Secondary outcomes may be analyzed with different levels of precision or power, but limitations should be acknowledged.
See whether sample size should be based only on the primary outcome.
What Determines Sample Size in Qualitative Research?
Qualitative sample adequacy is generally evaluated differently from conventional quantitative power or precision.
The relevant considerations include the research aim, specificity of selected cases, methodological tradition, quality of evidence, and analytical depth.
Malterud, Siersma, and Guassora's information power framework provides one approach for considering sample adequacy in qualitative interview studies.
Some qualitative traditions use forms of saturation, while others emphasize different criteria for determining whether sufficient evidence has been collected.
A narrowly focused study involving information-rich participants may require a different number of cases from a study examining several contrasting institutional contexts.
Researchers should avoid imposing conventional statistical power formulas on qualitative designs or assuming that a fixed number of interviews is universally adequate.
Can a Study Have Too Many Participants?
A larger sample is not inherently problematic, but unnecessary recruitment may create ethical, financial, and analytical burdens.
In intervention studies, recruiting substantially more participants than needed may expose additional people to research procedures without a corresponding scientific benefit.
In qualitative studies, an excessively large and heterogeneous dataset may become difficult to analyze meaningfully within the available resources.
However, the appropriate upper limit depends on the design and purpose.
Researchers should seek sufficient evidence for the intended conclusions rather than assume that maximizing participant numbers always improves quality.
How Should Sample Size Be Justified in the Methodology?
The justification should identify the primary objective, planned analysis, relevant assumptions, and basis for the recruitment target.
For quantitative studies, report the effect or precision target, significance level and power where applicable, variability assumptions, allocation, design adjustments, and expected missing data.
For qualitative studies, explain the sampling strategy, relevance of selected cases, and methodology-appropriate criteria for analytical adequacy.
If the sample size is constrained by feasibility, distinguish the feasible number from a statistically established requirement.
Where uncertainty exists, sensitivity analysis may help show how conclusions depend on the planning assumptions.
The strongest justification explains why the planned amount of evidence is sufficient for the specific research question rather than relying on a conventional participant count.
