03 · What You Need to Know
How AI Can Support Sample Size Planning Without Replacing Statistical Justification
What Does Sample Size Determination Actually Involve?
Sample size determination concerns how much information a study requires to address its research question with an appropriate degree of precision, statistical power, or other methodological adequacy.
In quantitative research, sample-size calculations may be based on estimating a population characteristic, testing a specified hypothesis, or achieving a desired level of precision for a particular parameter.
For example, estimating the prevalence of generative AI use among university students requires different inputs from estimating the difference in achievement between two instructional conditions.
In qualitative research, sample adequacy may instead depend on the study aim, methodological tradition, participant specificity, and richness of the information obtained.
AI should therefore begin by identifying the purpose of the study rather than immediately selecting a familiar formula.
Why Is Population Size Alone Insufficient?
A common AI response is to ask for the population size and then calculate a sample using a conventional confidence level and margin of error.
That approach may be relevant for estimating a population proportion under specified probability sampling assumptions.
However, population size alone does not determine the number of participants needed for an experiment, regression analysis, longitudinal study, or qualitative investigation.
For example, two studies may involve the same population of 5,000 students but require substantially different sample sizes because their objectives differ.
| Research objective |
Relevant planning consideration |
Why population size is insufficient |
| Estimate AI use prevalence |
Desired confidence level, precision, anticipated proportion, and sampling design. |
Precision depends on variability and the selection procedure. |
| Compare two teaching interventions |
Target effect, outcome variability, power, significance level, and allocation ratio. |
The detectable difference and its variability drive the calculation. |
| Estimate a correlation |
Expected association, desired power or precision, and analytical assumptions. |
The magnitude and uncertainty of the association matter. |
| Analyze clustered observations |
Number and size of clusters, intraclass correlation, and allocation structure. |
Observations within clusters may not provide independent information. |
| Conduct qualitative interviews |
Study aim, participant specificity, methodological approach, and information richness. |
Conventional survey precision formulas do not establish qualitative adequacy. |
AI recommendations should therefore be evaluated according to the particular statistical or methodological question being addressed.
What Is the Difference Between Precision-Based and Power-Based Sample Size Calculations?
Precision-based calculations determine how much information is needed to estimate a parameter within a desired degree of uncertainty.
Power-based calculations determine how much information is needed to achieve a specified probability of detecting a particular effect under a statistical testing framework.
Precision-based planning
Focuses on the desired uncertainty of an estimate, such as the width of a confidence interval for a population proportion.
Power-based planning
Focuses on the probability of rejecting a specified null hypothesis when an assumed alternative effect is present.
These approaches answer different questions.
For example, a descriptive survey may prioritize estimating AI use prevalence within a specified margin of error. A randomized experiment may prioritize detecting a meaningful difference in achievement between intervention conditions.
AI should not apply a descriptive survey formula to an experimental study merely because the researcher provides the population size.
What Inputs Are Needed for a Sample Size Calculation?
The required inputs depend on the selected method.
| Input |
Purpose |
Important consideration |
| Primary outcome or estimand |
Defines what the study intends to estimate or test. |
The calculation should correspond to the principal research objective. |
| Effect size or target precision |
Specifies the difference to detect or uncertainty to achieve. |
Should be justified rather than invented. |
| Significance level |
Specifies the Type I error criterion for a hypothesis test. |
Must reflect the planned testing framework. |
| Statistical power |
Specifies the desired probability of detecting the assumed effect. |
Common conventions are not substitutes for justification. |
| Outcome variability |
Influences the precision of estimated differences or parameters. |
May require prior studies or preliminary data. |
| Sampling or allocation design |
Determines how observations are selected or assigned. |
Clustering, stratification, and unequal allocation may affect requirements. |
| Expected missingness or attrition |
Accounts for anticipated loss of usable information. |
Requires plausible assumptions about participation and follow-up. |
AI should identify missing inputs and explain their implications rather than silently insert conventional values.
Can AI Calculate Sample Size for a Population Proportion?
Yes. AI can explain the conventional formula used to estimate a population proportion under simple random sampling assumptions.
For a sufficiently large population, a commonly used starting formula is:
Illustrative Formula
n₀ = Z² × p(1 − p) / e²
Where:
n₀ = initial required sample size.
Z = standard normal critical value corresponding to the desired confidence level.
p = anticipated population proportion.
e = desired absolute margin of error.
For example, using a 95% confidence level, an anticipated proportion of 0.50, and a margin of error of 0.05 gives an initial sample size of approximately 384.16, conventionally rounded upward to 385.
For a finite population of 5,000 units, applying the usual finite population correction gives approximately 357 completed observations after rounding upward.
However, this calculation assumes a suitable probability sampling design and does not automatically account for nonresponse, clustering, or other departures from the assumptions.
The number 357 is therefore not a universal requirement for every study involving 5,000 individuals.
Why Is the Anticipated Proportion Important?
For a population proportion, the variance depends on the anticipated value of the proportion.
When no reliable prior estimate is available, using p = 0.50 provides the maximum value of p(1 − p), making it a conservative choice for the conventional proportion formula.
However, this choice should not be confused with assuming that exactly half the population possesses the characteristic.
It is a planning assumption used to avoid understating variability when the true proportion is unknown.
AI should explain the assumption rather than present 0.50 as an observed population value.
Can AI Estimate Sample Size for an Experimental Study?
AI can help identify the inputs required for power-based calculations involving comparisons between experimental conditions.
For example, a study comparing mathematics achievement between students assigned to AI tutoring and usual instruction may require information about the minimum effect of interest, outcome variability, allocation ratio, significance level, and desired power.
The appropriate calculation also depends on whether observations are independent, paired, repeated, or clustered.
Cohen (1988) provides a foundational treatment of statistical power analysis and the relationship between effect size, significance level, power, and sample size.
However, AI should not assume a conventional effect size such as Cohen's d = 0.50 merely because it is often described as medium.
The target effect should reflect a scientifically or practically meaningful difference, supported by relevant evidence or a defensible rationale.
Why Should Researchers Be Cautious About AI-Selected Effect Sizes?
Effect size assumptions can substantially influence sample-size estimates.
For many statistical tests, smaller effects require larger samples to achieve the same statistical power under otherwise comparable conditions.
AI may recommend conventional small, medium, or large effect sizes without examining the educational context.
For example, a modest improvement in academic achievement may be educationally meaningful even if it falls below a conventional medium effect size.
Conversely, an assumed large effect may produce an unrealistically small required sample.
Researchers should justify the target effect using relevant literature, substantive considerations, or a clearly specified minimum effect of interest.
When uncertainty remains, sensitivity calculations across plausible effect sizes may be more informative than a single unsupported estimate.
Can AI Help Calculate Sample Size for Regression or Structural Equation Modeling?
AI can identify relevant planning considerations, but no universal participant-to-variable ratio establishes adequacy for every regression or structural equation model.
Requirements depend on the number and nature of parameters, anticipated effects, predictor distributions, measurement quality, missing data, model identification, and the intended inferential objective.
For example, a multiple regression study examining a primary predictor may require a different calculation from one designed to detect a small interaction effect.
Similarly, structural equation modeling requirements depend on the specified measurement and structural models rather than a single rule such as ten respondents per item.
AI-generated rules of thumb should therefore be treated cautiously and checked against appropriate design-specific methods.
How Does Cluster Sampling Affect Sample Size?
When observations are grouped within classrooms, schools, or institutions, their outcomes may be correlated.
As a result, a sample containing 300 students from a small number of classrooms may provide less independent information than 300 students selected independently across a broader population.
A design effect is sometimes used to represent the impact of the sampling design on estimator variance relative to a specified reference design.
For equal-sized clusters under a simple approximation, the design effect may be expressed as:
Illustrative Cluster Design Effect
DEFF = 1 + (m − 1)ρ
Where:
m = number of observations per cluster.
ρ = intraclass correlation coefficient.
This expression is an approximation under particular assumptions. Unequal cluster sizes, allocation structures, and analytical methods may require more appropriate calculations.
Researchers should not automatically multiply every sample-size estimate by an arbitrary design effect.
The specific requirements are examined further when considering AI-assisted design effect calculations.
Should Sample Size Be Adjusted for Attrition?
When some recruited participants are expected not to provide usable outcome data, researchers may need to recruit more individuals than the number required for the final analysis.
For example, if 200 completed observations are required and the anticipated attrition proportion is 20%, a simple adjustment gives:
Illustrative Attrition Adjustment
Required recruitment = 200 / (1 − 0.20) = 250 participants
This calculation assumes that the specified attrition proportion is applicable to the recruitment and follow-up process.
It does not establish that the missing data will be ignorable or that attrition-related bias has been eliminated.
Researchers should distinguish increasing recruitment targets from addressing the methodological consequences of missing data.
A more detailed examination of attrition adjustments should consider the relevant assumptions and study structure.
Can AI Determine Sample Size for Qualitative Research?
AI can explain approaches to qualitative sample adequacy, but conventional statistical power or survey margin-of-error formulas generally do not determine how many qualitative interviews are sufficient.
Qualitative sampling decisions depend on the research question, methodological tradition, participant characteristics, analytical strategy, and the richness of the information obtained.
Malterud, Siersma, and Guassora (2016) propose information power as one approach to evaluating sample adequacy in qualitative interview research.
Other qualitative traditions may emphasize theoretical sampling, saturation, or different forms of analytical sufficiency.
AI should not recommend a universal number such as 10, 15, or 30 participants without explaining the methodological rationale.
Researchers should examine qualitative sample adequacy separately from conventional quantitative sample-size calculations.
Can AI Perform the Arithmetic Correctly?
AI can explain formulas and perform calculations, but language-model outputs are not guaranteed to be numerically accurate.
Errors may arise from incorrect formula selection, substituted assumptions, rounding, inconsistent units, or mistakes in arithmetic.
Even a numerically correct calculation may be inappropriate if the formula does not correspond to the research design.
Researchers should independently reproduce important calculations using an appropriate statistical application, validated calculator, or transparent computational procedure.
The question of whether an AI chatbot's sample-size calculation should be trusted therefore involves both numerical verification and methodological evaluation.
Can AI Select the Correct Sample Size Formula?
AI may identify candidate formulas based on the research question and planned analysis.
However, selecting a formula requires understanding the estimand, outcome distribution, statistical test, sampling or allocation structure, and relevant assumptions.
For example, a formula for estimating a single population proportion is not automatically suitable for comparing two proportions or detecting a correlation.
Researchers should ask AI to explain why a particular formula applies and what information would make the recommendation inappropriate.
The more focused question of selecting an appropriate sample-size formula requires examining the intended analysis rather than matching keywords.
Why Is Sensitivity Analysis Useful During Sample Size Planning?
Sample-size calculations often depend on uncertain planning assumptions.
For example, researchers may not know the true standard deviation of achievement scores or the likely effect of an AI tutoring intervention.
Instead of relying on one assumed value, researchers can examine how the required sample changes across plausible scenarios.
AI can help organize such comparisons and explain which assumptions have the greatest influence on the result.
However, the ranges should be justified using relevant evidence or substantive reasoning rather than invented to make the study appear feasible.
Sensitivity analysis does not eliminate uncertainty, but it makes the consequences of uncertain assumptions more visible.
Can AI Determine Whether a Proposed Sample Size Is Feasible?
AI may compare the estimated requirement with the accessible population, recruitment period, anticipated participation, and available resources.
However, feasibility does not establish statistical adequacy.
If a defensible calculation indicates that 400 completed observations are needed but only 100 participants are accessible, the appropriate response is not automatically to reduce the requirement to 100.
The researcher may need to reconsider the precision objective, research question, design, recruitment strategy, or available resources.
Any modification should be justified and its consequences for inference made explicit.
Researchers should therefore evaluate the feasibility of the proposed design alongside the statistical requirements.
What Should a Defensible Sample Size Justification Include?
A sample-size justification should allow readers to understand how the number was obtained and why the assumptions are appropriate.
For quantitative studies, relevant documentation may include the primary outcome, estimand or hypothesis, calculation method, significance level or confidence level, target power or precision, effect size or variability assumptions, sampling design, attrition adjustment, and software used.
Not every element applies to every study. A descriptive precision calculation, for example, does not necessarily require a statistical power assumption.
For qualitative research, the justification should reflect the methodological tradition and the approach used to evaluate sample adequacy.
Researchers should avoid presenting a number without explaining the reasoning that produced it.
How Can You Use AI to Plan a Sample Size Calculation?
1. Define the primary research objective. Identify the main parameter, comparison, or methodological purpose requiring adequate information.
2. Specify the study design. Describe sampling, intervention allocation, repeated measurements, clustering, and other relevant features.
3. Identify the calculation approach. Determine whether precision-based, power-based, simulation-based, or another method is appropriate.
4. List required assumptions. Identify effect sizes, variability, significance levels, power, precision, and design parameters as applicable.
5. Verify input values. Obtain defensible estimates from relevant literature, preliminary data, or substantive justification.
6. Calculate and reproduce the result. Use an appropriate statistical procedure and independently verify the arithmetic.
7. Evaluate adjustments and sensitivity. Examine attrition, clustering, unequal allocation, and uncertainty in planning assumptions.
8. Document the justification. Report the method, inputs, assumptions, software, and final recruitment target where applicable.
This workflow is a practical planning approach rather than a validated AI-specific sample-size calculation procedure.
Watch Out
Do not accept a sample-size recommendation simply because AI provides a formula and a precise numerical result. The calculation may be arithmetically correct while relying on assumptions that do not apply to the research design.
