Can Generative AI Help Calculate a Design Effect for Sample Size?

If You Recruit 300 Students From Classrooms, Do You Really Have 300 Independent Observations?

You are planning a study evaluating AI-supported instruction across university classrooms. Your initial sample size calculation indicates that 300 students are needed.

However, instead of randomly selecting individual students from the university population, you plan to recruit entire classrooms. Students within each classroom share an instructor, learning environment, and instructional experiences.

You ask generative AI whether the original sample size is still sufficient. It recommends multiplying the number by a design effect of 1.5, producing a revised requirement of 450 students.

But where did 1.5 come from? Does the adjustment depend on the number of students per classroom? What if students within the same classroom have nearly identical outcomes, or what if the classrooms contain substantially different numbers of students?

A design effect is not simply an arbitrary multiplier added to make a sample size calculation appear more rigorous. It describes how a sampling design affects statistical variance relative to a specified reference design.

AI Can Calculate a Design Effect, but the Formula Must Match the Sampling Design

How to Calculate and Interpret Design Effects With AI Assistance

What Is a Design Effect?

A design effect, commonly abbreviated DEFF, expresses the variance of an estimator under a particular sampling design relative to its variance under a specified reference design, commonly simple random sampling.

Kish (1965) developed the concept as part of survey sampling methodology.

A design effect greater than one indicates that the actual design produces greater variance than the reference design for the estimator being considered.

A design effect below one indicates lower variance under the specified comparison.

Importantly, the design effect is not necessarily identical for every outcome or estimator in a study. Its interpretation depends on what is being estimated and how the sampling design operates.

Why Does Clustering Affect Statistical Precision?

Cluster sampling involves selecting groups of units, such as classrooms or schools, rather than independently selecting every individual from the entire population.

Students within the same classroom may share characteristics that influence their responses or outcomes.

For example, they may have the same instructor, receive similar instructional materials, or experience the same classroom conditions.

When observations within clusters are positively correlated, adding more individuals from the same cluster may provide less new information than adding the same number of independent individuals.

This can increase the variance of an estimate and influence the required sample size.

However, the consequences depend on the sampling design, the parameter being estimated, and the analytical method.

What Is the Intraclass Correlation Coefficient?

The intraclass correlation coefficient, or ICC, describes the similarity of observations within clusters under a specified statistical model.

In a simple random-intercept model, it can be understood as the proportion of total outcome variance attributable to differences between clusters.

For example, an ICC of 0.05 indicates that approximately 5% of the modeled outcome variance is attributable to between-cluster variation under that model.

Although 0.05 may appear small, its consequences can become substantial when many participants are recruited from each cluster.

AI may explain the ICC correctly while recommending an unsupported numerical value.

Researchers should obtain relevant estimates from comparable studies, preliminary data, or appropriately justified sensitivity scenarios.

What Is the Familiar Cluster Design Effect Formula?

For equal-sized clusters under a simplified exchangeable correlation structure, a commonly used approximation is:

The formula illustrates how cluster size and within-cluster correlation affect the variance inflation associated with clustering.

It should not be applied automatically to every clustered, stratified, or multistage sampling design.

Unequal cluster sizes, weighting, finite populations, and complex analytical structures may require different calculations.

How Do You Calculate a Design Effect Step by Step?

Suppose a researcher plans to recruit students from classrooms containing approximately 25 students each.

For illustration, assume an ICC of 0.05.

Under these assumptions, the approximate variance inflation relative to the reference design is 2.20.

If an appropriate simple random sampling calculation required 200 observations, a preliminary adjustment would be:

200 × 2.20 = 440 observations.

However, this multiplication is an approximation. The researcher must verify whether the reference calculation, clustering assumptions, and planned analysis make the adjustment appropriate.

Does a Design Effect of 2.20 Mean the Sample Is Twice as Biased?

No. Design effect concerns variance relative to a reference sampling design, not the magnitude of bias.

A design effect of 2.20 indicates that the variance of the relevant estimator is approximately 2.20 times the reference variance under the assumed conditions.

It does not imply that the estimate is 120% more biased or that the study is scientifically invalid.

Bias and variance are different statistical properties.

Researchers should therefore avoid describing a design effect as a direct measure of sampling bias or representativeness.

How Does Cluster Size Influence the Design Effect?

Under the simplified formula, larger cluster sizes increase the design effect when the ICC is positive.

Consider an ICC of 0.05.

These values are hypothetical and assume equal cluster sizes and the same ICC.

The example illustrates why recruiting more students from a small number of classrooms may be less statistically efficient than recruiting students across more classrooms.

However, practical decisions about the number and size of clusters must also consider recruitment, cost, and the intended design.

How Does the ICC Influence the Design Effect?

Holding cluster size constant, higher positive ICC values increase the design effect under the simplified model.

Suppose each classroom contains 25 students.

The required sample size can therefore be sensitive to the ICC assumption.

AI should not select a convenient ICC merely to produce an achievable recruitment target.

When relevant empirical estimates are uncertain, researchers should consider sensitivity analyses across defensible values.

What Is Effective Sample Size?

Effective sample size is sometimes used to express the amount of information in a complex sample relative to a specified reference design.

For an estimator with a design effect greater than zero, a simple variance-equivalent approximation is:

For example, if 440 observations have a design effect of 2.20:

n_effective = 440 / 2.20 = 200.

This suggests that the estimator has approximately the variance expected from 200 observations under the specified reference design.

However, effective sample size is estimator-dependent and should not be treated as a universal count of independent participants.

It also does not automatically establish adequate statistical power for every analysis.

Does the Number of Clusters Matter Separately From the Design Effect?

Yes. The number of independent clusters can be critical, especially in cluster-randomized trials.

For example, suppose a researcher has access to only two classrooms, with one assigned to AI tutoring and the other to usual instruction.

Recruiting more students within those two classrooms does not provide additional independently randomized classrooms.

The treatment condition would be completely confounded with classroom membership, making conventional cluster-level treatment-effect inference particularly problematic.

Hemming and colleagues (2011) discuss sample size considerations for cluster-randomized trials with a fixed number of clusters.

Researchers should therefore not assume that multiplying a participant count by a design effect resolves limitations arising from too few clusters.

What Happens When Cluster Sizes Are Unequal?

Real classrooms and schools often contain different numbers of participants.

Unequal cluster sizes can affect variance and statistical power, particularly when cluster size is related to outcomes or selection probabilities.

Under certain simplifying assumptions, an approximation incorporating the coefficient of variation of cluster size is:

This is an approximation for particular settings rather than a universal formula for unequal clusters.

Its suitability depends on the sampling or trial design, estimator, and relevant assumptions.

Researchers should not use it without verifying that those conditions are appropriate.

Can Stratification Reduce the Design Effect?

Yes. Stratification can improve statistical precision when the strata are appropriately defined and related to the study variable.

For example, stratifying a university survey by academic college may improve estimation efficiency under suitable allocation and weighting conditions.

However, stratification does not always reduce variance. Its effects depend on the population structure, allocation, and estimator.

AI should therefore not assume that every complex sampling design has a design effect greater than one.

Design effects can be below one when the actual design is more efficient than the reference design for the relevant estimator.

Can Unequal Sampling Weights Increase the Design Effect?

Yes. Unequal weights can increase estimator variance under certain conditions.

Kish's familiar weighting approximation is sometimes expressed as:

However, this approximation relies on assumptions about the relationship between weights and the study variable.

It should not be treated as an exact or universally appropriate measure of weighting-related variance inflation.

Researchers should use methods consistent with the actual sampling design and estimator.

Can AI Simply Multiply Different Design Effects Together?

AI may recommend multiplying a clustering design effect by a weighting design effect to obtain a combined adjustment.

Such approximations may be useful under particular conditions, but they are not universally valid.

Clustering, stratification, and weighting may interact, and the combined variance effect depends on the design and estimator.

Researchers should not assume that separate adjustments can always be multiplied without examining the underlying assumptions.

For complex survey designs, design-based variance estimation or appropriate simulation may provide a more defensible assessment.

Does a Design Effect Apply to Every Sample Size Calculation?

No. A design effect is relevant when the sampling or allocation structure changes estimator variance relative to the reference design.

For example, a simple random sample of individuals may not require a cluster design effect.

A cluster-randomized trial may require a design-specific power calculation that accounts for the number of clusters, cluster size, ICC, and analytical approach.

Similarly, stratified sampling or unequal weighting may require more detailed variance considerations.

AI should identify the actual design before recommending any multiplier.

Should Design Effect Be Applied Before Attrition Adjustment?

For a simple planning approximation, researchers may first adjust the required sample for the sampling design and then increase the recruitment target for anticipated attrition.

For example, suppose an initial simple random sampling calculation requires 200 completed observations, the appropriate design effect is 2.20, and anticipated attrition is 20%.

However, the calculation assumes that the design effect remains appropriate after accounting for the expected completed cluster sizes.

If attrition changes the number or size of clusters, the design effect itself may change.

For complex designs, clustering and attrition should therefore be considered jointly rather than treated as unrelated multipliers.

Can AI Calculate the Design Effect Correctly?

AI can explain formulas and perform arithmetic, but errors remain possible.

It may confuse cluster size with the number of clusters, use an unsupported ICC, apply the formula to an inappropriate design, or incorrectly interpret the result.

For example, substituting the total number of students for m in the equal-cluster formula would produce an incorrect adjustment.

Researchers should independently reproduce the calculation and verify that the inputs correspond to the intended sampling structure.

The broader principles of verifying AI-generated sample size calculations apply equally to design effects.

How Can You Audit an AI-Generated Design Effect?

The resulting design effect should be interpreted as a property of the specified design and estimator, not as a universal penalty for conducting clustered research.

Adjusting a Sample Size for Students Recruited From Classrooms

Under the simplified assumptions, recruiting 18 classrooms with approximately 25 completed observations each would yield around 450 observations.

However, the researcher must verify that the ICC is appropriate, classroom sizes are sufficiently similar, and the selected classrooms cover the intended student population.

The analysis must also account for the actual sampling design.

The decision: The researcher uses the design effect as a justified planning approximation rather than accepting AI's arbitrary multiplier.

If the actual classroom sizes vary substantially or the design includes additional stages or weights, a more appropriate design-specific calculation may be required.

The example demonstrates why sample size planning must reflect how observations are selected, not merely how many participants are expected.

Common Mistakes in AI-Generated Design Effect Calculations

Ask AI to Explain the Variance Structure Before Applying a Multiplier

A useful design effect calculation begins by identifying how observations are selected or assigned and whether the data contain clustering, stratification, or unequal weights.

AI should explain which reference design is being used and why the selected approximation applies.

A Prompt for Calculating a Design Effect

After reviewing the result, researchers should verify the underlying sample size assumptions and consider whether attrition adjustments are also necessary.

The final sample size justification should document the design effect, its assumptions, and the statistical method used to account for the actual sampling structure.

Before Accepting an AI-Generated Design Effect

Frequently Asked Questions About AI and Design Effects

A Design Effect Must Reflect the Sampling Design, Not a Convenient Multiplier

How to Cite This Guide

This guide is intended to be read, shared, and used in research, teaching, and academic work. If you draw on its ideas, explanations, or other content, please acknowledge the source by citing the guide. Doing so gives appropriate credit and helps your readers locate the original resource.