03 · What You Need to Know
Understanding Dependence Before Calculating Sample Size
What Is Clustering in Research?
Clustering occurs when observations are grouped within higher-level units and may be more similar to one another than to observations from other groups.
Examples include students within classrooms, teachers within schools, patients within hospitals, and employees within organizations.
Members of the same cluster may share environmental conditions, institutional practices, leadership, resources, or other characteristics.
Consequently, their outcomes may be correlated.
The extent and implications of clustering depend on the study design, outcome, and source of dependence. Researchers should not assume that every grouped dataset has the same correlation structure.
What Are Repeated Measures?
Repeated measures occur when the same participant or unit is observed on multiple occasions or under multiple conditions.
For example, students may complete writing assessments at baseline, midsemester, and the end of a course.
Measurements from the same student are usually related because they reflect characteristics that persist across occasions.
Repeated observations can provide information about change, trajectories, and within-person variation.
However, they do not represent independent participants. The analytical and sample size implications depend on the research question, measurement timing, correlation, and expected missingness.
Why Does Independence Matter for Sample Size?
Statistical precision depends on the amount of information available for estimating the target quantity.
When observations are independent, each observation may contribute relatively distinct information under the assumed model.
When observations are correlated, some information overlaps.
For example, students taught by the same instructor may respond similarly to an educational intervention because they share instructional conditions.
Counting these students as independent observations can underestimate uncertainty if the analysis ignores relevant clustering.
Sample size planning should therefore distinguish the number of recorded observations from the information available for the intended statistical comparison.
What Is the Intraclass Correlation Coefficient?
The intraclass correlation coefficient, commonly abbreviated ICC, describes the similarity of observations within clusters under a specified statistical model.
In a simple random-intercept model, it can be interpreted as the proportion of total outcome variance attributable to differences between clusters.
An ICC close to zero indicates relatively little within-cluster similarity under that model, while a larger positive ICC indicates greater similarity.
However, ICC values depend on the outcome, population, measurement conditions, and statistical model.
Researchers should not borrow an ICC from an unrelated study merely because the studies involve the same type of institution.
What Is the Design Effect?
The design effect describes how a sampling or study design changes the variance of an estimator relative to a specified reference design.
For equal-sized clusters under a simple exchangeable correlation model, a commonly used approximation is:
For example, if each cluster contains 30 individuals and the ICC is 0.05, the approximate design effect is 2.45.
This indicates that the variance under the simplified clustered design is approximately 2.45 times the reference variance under the stated assumptions.
However, this formula does not fully represent unequal cluster sizes, covariate adjustment, small numbers of clusters, or every cluster randomized design.
What Is Effective Sample Size?
Effective sample size is a way of expressing how much information a dependent or otherwise complex sample provides relative to a defined independent-sampling reference.
Under the simplified cluster design effect approximation, researchers sometimes calculate:
This calculation can illustrate the loss of precision associated with positive clustering.
However, effective sample size is not a universal property of a dataset. It depends on the estimator, outcome, and reference design.
Researchers should not use this approximation as a substitute for a sample size calculation tailored to the actual trial or observational design.
Why Does the Number of Clusters Matter?
The number of clusters can be particularly important when treatment is assigned or important exposures vary at the cluster level.
For example, a study randomizing 10 schools has only 10 independent randomization units, even if thousands of students are observed within them.
Increasing the number of students per school may improve precision, but its benefit can diminish as within-school correlation increases.
Adding independent schools may provide information that cannot be obtained simply by recruiting more students from existing schools.
Small numbers of clusters can also complicate variance estimation and statistical inference.
Researchers should therefore plan both the number of clusters and the number of participants within clusters.
Is Recruiting More Participants Within Each Cluster Always Helpful?
Additional participants can improve precision, but the marginal benefit depends on the ICC, cluster size, and analysis.
When within-cluster correlation is substantial, adding many individuals to a small number of clusters may be less efficient than increasing the number of clusters.
However, recruiting additional clusters may be more expensive or logistically difficult.
The appropriate allocation should consider statistical efficiency, recruitment capacity, intervention delivery, and available resources.
There is no universal rule that all studies should prioritize cluster count over individual count.
How Do Unequal Cluster Sizes Affect Sample Size?
Cluster sizes are often unequal because institutions differ in enrollment, recruitment capacity, or participant eligibility.
Unequal cluster sizes can reduce statistical efficiency under common models, although the extent depends on the allocation, correlation structure, and estimator.
Researchers should consider the anticipated distribution of cluster sizes rather than use only the average.
The coefficient of variation in cluster size is sometimes incorporated into design effect approximations.
However, the appropriate formula depends on the study design and analysis.
Simulation may be useful when cluster sizes vary substantially or the design includes several complex features.
Does Clustering Matter Only in Cluster Randomized Trials?
No. Clustering can affect observational studies, surveys, quasi-experiments, and individually randomized trials.
For example, students individually randomized within classrooms may still have correlated outcomes because of shared classroom conditions.
Likewise, observational studies comparing students across schools may need to account for institutional clustering.
However, the implications differ depending on whether treatment assignment, exposure, or outcome variation occurs within or between clusters.
Researchers should not automatically apply the same sample size inflation factor to every study containing grouped observations.
Can Repeated Measures Improve Statistical Power?
Yes, under appropriate conditions.
Repeated measurements may improve precision by allowing researchers to account for baseline variation or estimate within-person changes.
For example, an analysis of final writing performance adjusted for baseline writing score may be more precise than an unadjusted comparison when baseline scores are strongly predictive of final outcomes.
However, the benefit depends on the correlation structure, outcome variability, timing, and statistical model.
Repeated measurements do not automatically increase power in proportion to the number of observations.
Additional measurements may also increase participant burden and missing-data risk.
How Does Within-Person Correlation Affect Sample Size?
Within-person correlation influences the variance of changes, repeated-measures estimates, and other longitudinal quantities.
For two measurements with equal variance, the variance of their difference depends on the correlation between them.
When the correlation is positive, the variance of a paired difference can be smaller than it would be under independence.
However, this formula assumes equal variances and concerns a particular paired comparison.
It should not be generalized directly to every longitudinal model or intervention design.
Is a Pretest-Posttest Design Always More Efficient?
Not necessarily. The efficiency of a pretest-posttest design depends on the research question, baseline-outcome relationship, measurement reliability, and analysis.
For randomized studies, baseline-adjusted analysis of covariance can often provide greater precision than an unadjusted final-score comparison when the baseline measure is predictive.
However, analyzing change scores and adjusting final scores for baseline are not mathematically or inferentially identical in every setting.
The appropriate method should be selected during design and reflected in sample size calculations.
Researchers should not choose between methods after examining which produces the most favorable result.
What If Participants Are Measured More Than Twice?
Studies with several measurement occasions may investigate trajectories, time-by-treatment interactions, or other longitudinal patterns.
Sample size planning must reflect the intended analytical target.
For example, detecting a difference in final outcomes is not the same as detecting a difference in rates of change over time.
Repeated-measures models may require assumptions about the covariance structure, measurement intervals, and expected dropout.
Additional measurement occasions may improve precision under some conditions, but the benefit can diminish when measurements are highly correlated.
Researchers should justify measurement frequency according to the scientific question rather than assume that more observations always provide proportionately more information.
How Does Attrition Affect Repeated-Measures Sample Size Planning?
Longitudinal studies may lose observations because participants withdraw, miss assessments, or become unavailable.
Attrition can reduce statistical power and introduce bias when missingness relates to unobserved outcomes.
Researchers should consider anticipated retention when calculating recruitment targets.
However, simply increasing the initial sample does not eliminate bias caused by informative missingness.
Retention procedures, relevant baseline measurements, and appropriate missing-data methods should be planned together.
The principles of missing-data planning are therefore particularly relevant to repeated-measures designs.
What If the Study Has Both Clustering and Repeated Measures?
Many educational, healthcare, and organizational studies involve multiple sources of dependence.
For example, students may be measured repeatedly while remaining nested within classrooms and schools.
The resulting data structure may include correlations among repeated observations from the same student and among students from the same classroom.
Sample size planning should account for the levels relevant to the intended inference.
Depending on the design, multilevel models, marginal models, or simulation-based calculations may be appropriate.
Researchers should not assume that a single ICC or simple design effect captures every source of dependence.
How Should Sample Size Be Calculated for Multilevel Studies?
The calculation should correspond to the intended estimator, allocation structure, and outcome model.
Relevant inputs may include the number of clusters, anticipated cluster sizes, ICCs, outcome variability, effect size, allocation ratio, and relevant covariate relationships.
For longitudinal designs, measurement occasions, within-person correlations, and expected missingness may also be necessary.
Analytical formulas can be appropriate for relatively simple designs.
For more complex structures, simulation-based power or precision analysis may provide a useful alternative.
However, simulation results depend on the assumptions used to generate the data and the correctness of the simulated analysis procedure.
Where Should ICC and Correlation Assumptions Come From?
Researchers may use relevant previous studies, pilot data, institutional records, or carefully justified scenarios.
However, estimates from small pilot studies may be imprecise.
ICC values may also differ across outcomes, populations, and institutional contexts.
Rather than rely on one apparently precise value, researchers may examine sample size requirements across a plausible range.
The source and uncertainty of assumptions should be documented.
Sensitivity analysis of planning assumptions can help determine whether the proposed design remains feasible under less favorable conditions.
Should Sample Size Planning Focus Only on Statistical Power?
No. Sample size may also be determined by the precision required for estimation, the number of clusters needed for valid inference, or other design-specific objectives.
For example, a descriptive multilevel study may prioritize sufficiently precise estimates of between-school variation rather than a conventional hypothesis test.
Researchers should define whether the goal is hypothesis testing, estimation, prediction, or another analytical purpose.
The relevant planning criterion should follow that purpose.
Power calculations are not interchangeable with precision-based calculations, and neither guarantees that the resulting design is free from bias.
What Happens If the Number of Clusters Is Too Small?
A small number of clusters may limit precision and complicate inference about cluster-level effects or variation.
For example, a study involving four schools may not support reliable estimation of a complex model containing several school-level predictors.
Some methods also rely on large-sample approximations that perform poorly with few clusters.
Small-sample corrections or alternative procedures may be considered, but they do not create information equivalent to having additional independent clusters.
Researchers should assess whether the intended inferential claims are feasible before collecting data.
Where necessary, the research question or design may need to be revised.
Can Researchers Ignore Clustering If the ICC Is Small?
Not automatically. Even a modest ICC may substantially affect precision when clusters are large.
For example, an ICC of 0.02 may appear small, but its contribution to the design effect increases with cluster size under the simplified approximation.
The importance of clustering depends on the design, estimator, number of clusters, and intended inference.
Researchers should not decide to ignore dependence solely because an estimated ICC fails to reach statistical significance.
The appropriate analysis should reflect the study structure and relevant uncertainty.
How Should Clustering and Repeated Measures Be Reflected in the SAP?
The statistical analysis plan should describe the hierarchy or longitudinal structure, the analytical unit, the allocation unit where relevant, and the method for accounting for dependence.
It should also identify the target effect, relevant covariates, covariance assumptions, and uncertainty estimation procedures.
For example, a classroom randomized study should distinguish the classroom as the randomization unit from the student as the outcome measurement unit.
The analysis and sample size assumptions should be consistent.
Researchers should avoid planning an independent-samples analysis when the design requires an approach that accounts for clustered allocation.
When Should a Statistician Be Involved?
Statistical expertise is particularly valuable before recruitment when the study includes cluster randomization, several hierarchical levels, unequal cluster sizes, or complex longitudinal measurements.
A statistician can help identify the target estimand, determine the required planning parameters, and evaluate feasible analytical approaches.
Early involvement may reveal that the proposed number of schools, classrooms, or measurement occasions is insufficient.
However, methodological consultation should also consider practical and substantive knowledge from the research setting.
The objective is to develop a design that is both scientifically defensible and operationally feasible.
Watch Out
Do not equate the total number of recorded observations with the number of independent statistical units. Three hundred students from ten classrooms and 300 independently sampled students may provide different amounts of information. Repeatedly measuring the same participants does not multiply the independent sample size by the number of measurement occasions.
