03 · What You Need to Know
How Generative AI Calculates Research Sample Sizes
What Does a Sample Size Calculation Actually Determine?
A sample size calculation estimates how much information a study may need to achieve a specified statistical objective under stated assumptions.
That objective might be detecting a particular effect with a chosen statistical power, estimating a population parameter with a desired level of precision, or obtaining sufficient information for a planned model.
These objectives are not interchangeable. A sample size appropriate for estimating a population proportion within a particular margin of error may not be adequate for detecting a difference between two groups.
Likewise, a study designed to estimate an overall mean may require a different sample size from one intended to compare several subgroups.
Lakens (2022), in Sample Size Justification, explains that researchers can justify sample sizes through several approaches, including power analysis, precision-based planning, resource constraints, and near-complete population measurement. The central issue is whether the planned sample can provide useful information for the research objective. The published review provides a detailed discussion of these approaches.
AI should therefore begin by asking what the researcher wants the sample to accomplish rather than immediately applying a formula.
What Information Does AI Need Before Calculating Sample Size?
The necessary inputs depend on the analytical objective and statistical method.
| Input |
What it means |
Why it matters |
| Research design |
Whether observations are independent, paired, repeated, clustered, or otherwise structured. |
Determines how information accumulates and which calculation is appropriate. |
| Primary outcome |
The main variable or endpoint the study intends to analyze. |
Different outcome types may require different statistical models. |
| Primary analysis |
The statistical procedure planned for the main research question. |
Power and precision depend on the estimator or test. |
| Effect size |
The effect magnitude the study is designed to detect or evaluate. |
Smaller target effects generally require more information. |
| Significance level |
The prespecified Type I error threshold for a hypothesis test. |
Affects the rejection criterion and required sample size. |
| Statistical power |
The probability of rejecting the null hypothesis under a specified alternative and model. |
Higher desired power generally increases the required sample size. |
| Allocation ratio |
The planned relative sizes of comparison groups. |
Unequal allocation can change statistical efficiency. |
| Attrition or missingness |
Expected loss of usable observations. |
May require recruiting more participants than the analyzable sample target. |
Specialized designs may require additional parameters. For example, cluster-randomized studies need information about cluster sizes and intraclass correlation, while longitudinal studies may require assumptions about correlations between repeated measurements.
When these inputs are missing, AI should identify the uncertainty instead of silently choosing convenient defaults.
What Is Statistical Power, and Why Does It Affect Sample Size?
Statistical power is the probability that a hypothesis test rejects its null hypothesis under a specified alternative hypothesis and statistical model.
For example, a study planned with 80% power to detect a particular effect has an 80% probability of obtaining a statistically significant result under the assumed effect and model, when the other design assumptions hold.
It does not mean that there is an 80% probability the research hypothesis is true or that the study has an 80% chance of producing scientifically important findings.
Power is conditional on assumptions about the effect magnitude, variability, statistical procedure, and other design features.
Researchers often use 80% or 90% as planning targets, but these are conventions rather than universal requirements. The appropriate target depends on the consequences of missed effects, the research purpose, and resource constraints.
AI should not assume that 80% power is automatically sufficient for every study.
Why Is the Assumed Effect Size So Important?
The assumed effect size is often among the most influential inputs in a power-based sample size calculation.
For a simple comparison of two independent means, detecting a smaller standardized difference generally requires more participants than detecting a larger one, holding other conditions constant.
Suppose a researcher expects AI-assisted instruction to produce a standardized mean difference of 0.50. That assumption will lead to a different sample size from an assumed difference of 0.20.
Neither value should be chosen merely because it corresponds to a familiar small, medium, or large effect-size convention.
Ideally, the planning effect should be informed by relevant prior research, a substantively meaningful minimum effect, pilot evidence interpreted cautiously, or another defensible justification.
Lakens (2022) emphasizes the importance of considering the smallest effect size of interest and the informational value of the proposed study.
AI may help calculate or explain standardized effect sizes, but the researcher must justify which effect magnitude should inform the sample size calculation.
Can AI Calculate Sample Size for an Independent-Samples t-Test?
Yes. When the design and parameters are specified, a computational AI system can calculate sample size using statistical power functions.
For a conventional equal-allocation comparison of two independent means, a useful normal-approximation formula illustrates how the main inputs influence the required sample size.
This is an approximate planning calculation, not an exact noncentral-t power analysis. An exact calculation for the conventional pooled-variance independent-samples t-test may require a slightly larger sample.
For these inputs, a standard noncentral-t calculation gives approximately 64 participants per group, or 128 analyzable participants in total.
The distinction matters because AI may present a normal approximation as though it were the exact result from statistical software.
Official statsmodels documentation identifies the effect size, group allocation ratio, significance level, power, and test alternative as inputs to its independent-samples t-test power calculation.
Why Does a Paired Study Need a Different Sample Size Calculation?
In a paired study, the same participants are measured more than once or observations are otherwise meaningfully matched.
The relevant variability for a paired mean comparison is the standard deviation of the within-pair differences.
Two studies may have the same pretest and posttest means and standard deviations but different correlations between measurements. Their paired-difference variability, and therefore their required sample sizes, can differ.
For example, highly correlated pretest and posttest scores may produce relatively precise estimates of within-participant change when the correlation is positive.
AI should therefore not calculate sample size for a paired study using an independent-groups formula without justification.
Researchers must establish whether observations are paired or independent and specify the relevant effect-size definition.
A calculation based on the wrong observation structure may produce a precise-looking but inappropriate recruitment target.
Can AI Calculate Sample Size for Correlation or Regression?
AI can assist with sample size planning for correlation and regression, but these analyses require different assumptions from a simple comparison of means.
For correlation, a power calculation may depend on the population correlation the researcher wants to detect, the significance level, the test alternative, and the desired power.
For regression, the required sample size depends on the specific inferential objective. Testing one regression coefficient is not necessarily equivalent to testing an entire model.
Logistic regression introduces additional considerations, including the outcome frequency, predictor distributions, model complexity, and potential separation.
Prediction-model development also requires attention to overfitting, expected model performance, and the intended validation strategy.
Rules such as recruiting a fixed number of participants per predictor are not universal substitutes for a study-specific justification.
AI should first identify the regression model and analytical objective before recommending a sample size method.
Why Can AI Underestimate Sample Size for Clustered Studies?
Clustered studies involve observations grouped within classrooms, schools, hospitals, or other units.
Participants within a cluster may be more similar than independently sampled individuals. As a result, the information provided by a clustered sample may be lower than that of an equally sized independent sample for some estimators.
A commonly used approximation for equal-sized clusters is:
For estimators and designs to which the approximation applies, the variance may be approximately 2.2 times the corresponding independent-sampling variance.
This does not mean that every cluster-randomized study can obtain its final sample size simply by multiplying an individual-level calculation by 2.20.
Cluster-randomized trials also require attention to the number of clusters, allocation, cluster-size variation, and the level at which treatment is assigned.
Researchers should ensure that AI recognizes repeated-measures or clustered data before accepting a sample size recommendation.
Can AI Calculate Sample Size for a Survey Using a Margin of Error?
Yes. Precision-based planning is appropriate when the objective is to estimate a population parameter within a specified margin of error.
For a population proportion under simple random sampling, a commonly used normal-approximation formula is:
This familiar result is not a universal minimum sample size for quantitative research.
It applies to a particular precision objective and sampling approximation. It does not establish adequate power for group comparisons, regression coefficients, or other hypotheses.
When sampling from a relatively small finite population without replacement, a finite-population correction may be relevant. Complex sampling designs may require additional adjustments.
Furthermore, a nominal 5% margin of error does not account for selection bias or systematic nonresponse. Increasing sample size alone cannot eliminate those problems.
How Should AI Account for Attrition and Nonresponse?
A study may need to recruit more participants than the number required for its final analysis.
Suppose the power calculation requires 128 participants with usable outcome data. If the researcher expects 20% attrition, the initial recruitment target should account for that loss.
This adjustment does not guarantee that 128 usable observations will remain. Actual attrition may differ from expectations, and missingness can introduce bias even when the final sample size is adequate.
For studies with unequal attrition across groups or complex longitudinal missingness, more detailed planning may be necessary.
AI should distinguish the analyzable sample target from the number initially recruited.
Can AI Calculate Sample Size Without an Expected Effect Size?
Sometimes, depending on the planning objective.
A power analysis for detecting a specified effect generally requires an assumed effect magnitude or an equivalent alternative-hypothesis specification.
However, researchers may instead justify sample size through a precision target, resource constraints, or another approach appropriate to the scientific objective.
For example, a descriptive study might prioritize estimating a population proportion with a particular margin of error rather than testing a difference between groups.
If the effect size is uncertain, researchers can perform sensitivity analyses across plausible values instead of inventing a single precise assumption.
Lakens (2022) argues that sample size justification should focus on what the collected data can meaningfully establish, rather than treating power analysis as the only legitimate approach.
AI should therefore explain which planning methods remain available when reliable effect-size information is limited.
Can AI Use G*Power or Other Sample Size Software?
AI can explain how to configure a sample size calculation in G*Power and can generate code for comparable calculations in R or Python.
However, an AI system should not claim to have executed G*Power unless it actually has access to that application and performed the computation.
Different programs may implement different test formulations or numerical methods. For example, an approximate normal-theory calculation may not exactly match a noncentral-t calculation.
Researchers should record the software, procedure, assumptions, and settings used.
When AI generates Python code using statsmodels, its power-analysis functions can calculate sample sizes for supported tests. The official documentation describes the relevant parameters and numerical solution procedures.
For specialized designs, researchers may need dedicated software, simulation-based power analysis, or statistical consultation.
Why Can AI Produce Different Sample Sizes From the Same Research Question?
Two AI responses may produce different numbers because they make different assumptions about the analysis.
One response may assume a medium effect size, while another assumes a small effect. One may use a one-sided test, while another uses a two-sided test.
Other differences may arise from group allocation, desired power, attrition adjustments, clustering, or whether the calculation uses an approximation or a more exact numerical method.
The numbers may each be internally consistent with their assumptions while answering different planning questions.
Researchers should not compare sample size recommendations without first comparing their inputs.
A defensible calculation should make every consequential assumption visible.
Does a Correct Sample Size Guarantee a Successful Study?
No. A sample size calculation establishes expected statistical properties under specified assumptions. It does not guarantee statistical significance, accurate measurement, representative sampling, or valid causal conclusions.
A study can meet its planned sample size and still suffer from selection bias, poor measurement, substantial protocol deviations, or model misspecification.
Likewise, a study with fewer participants than planned may still provide useful information, although its precision or power may differ from the original expectations.
Sample size should be understood as one component of research design quality rather than a numerical certificate of validity.
How Should Researchers Verify AI-Generated Sample Size Calculations?
Verification begins with the assumptions, not the final number.
Confirm the primary outcome, intended statistical analysis, effect size or precision target, significance level, power, allocation ratio, and observation structure.
Then reproduce the calculation using independently reviewed statistical software or a documented numerical method.
For consequential studies, independent verification of AI-generated statistical calculations should include the exact settings and any adjustments for clustering, attrition, or multiple primary analyses.
Agreement between two programs is useful evidence of computational consistency, but it does not establish that the assumptions are scientifically justified.
