03 · What You Need to Know
What to Verify Before Accepting an AI-Generated Sample Size
Why Is Independent Verification Necessary?
Generative AI can explain statistical concepts, recommend calculation procedures, and perform mathematical operations. However, its responses may contain methodological or computational errors.
A chatbot may correctly evaluate a formula that does not apply to the study. It may also introduce assumptions that the researcher never supplied.
For example, AI may recommend a sample size for estimating a population proportion when the research actually involves testing a difference between two intervention groups.
The resulting number may be mathematically reproducible but scientifically irrelevant to the primary analysis.
Lakens (2022) emphasizes that sample size justification should reflect the study's inferential goals and the rationale for the information required.
Independent verification is therefore necessary to establish both computational correctness and methodological appropriateness.
What Are the Main Components of a Sample Size Audit?
A complete audit should examine the statistical question, calculation method, assumptions, design adjustments, numerical implementation, and final interpretation.
| Verification area |
What researchers must check |
Potential AI-related error |
| Research objective |
Does the calculation address the primary estimand or hypothesis? |
AI calculates a survey sample for an experimental comparison. |
| Formula selection |
Does the method correspond to the outcome and planned analysis? |
AI applies a familiar equation to an incompatible statistical problem. |
| Input assumptions |
Are effect sizes, variability, power, and precision justified? |
AI silently inserts conventional values. |
| Sampling and allocation |
Does the calculation reflect clustering, stratification, or unequal groups? |
AI assumes independent observations or equal allocation. |
| Design adjustments |
Are finite population corrections and design effects appropriate? |
AI applies an arbitrary multiplier or unjustified correction. |
| Attrition |
Is anticipated loss appropriately estimated and incorporated? |
AI adds a dropout percentage incorrectly. |
| Arithmetic |
Are substitutions, calculations, units, and rounding correct? |
AI produces an incorrect numerical result. |
| Interpretation |
What does the calculated sample size actually support? |
AI presents a conditional estimate as a universal requirement. |
These components should be examined together. A calculation that passes one check may still fail another.
First, Verify the Primary Research Objective
Before checking any formula, determine exactly what the study intends to estimate or test.
For example, a study may seek to estimate the prevalence of generative AI use among undergraduate students.
Another may investigate whether AI tutoring improves mathematics achievement compared with usual instruction.
A third may examine whether academic self-efficacy predicts intention to use generative AI after accounting for selected covariates.
These questions require different sample size considerations.
AI should not select a calculation merely because the researcher provides the total population size or mentions a familiar statistical method.
The first verification question is whether the calculation addresses the study's primary estimand, hypothesis, or precision objective.
Second, Verify the Sample Size Formula
Once the statistical objective is clear, researchers should examine whether the selected formula or computational procedure is appropriate.
For example, a conventional formula for estimating a population proportion is:
Illustrative Proportion Formula
n₀ = Z² × p(1 − p) / e²
Where:
Z = normal critical value for the selected confidence level.
p = anticipated population proportion.
e = desired absolute margin of error.
This expression may be appropriate for a descriptive population proportion estimate under specified simple random sampling assumptions.
However, it does not automatically apply to experimental comparisons, regression coefficients, correlations, or complex longitudinal analyses.
Researchers should independently verify the appropriateness of the sample size formula before accepting its result.
Third, Verify the Effect Size Assumption
For power-based calculations, the assumed effect size can substantially influence the required sample.
AI may recommend Cohen's d = 0.50 because it is conventionally described as a medium standardized effect.
However, this classification does not establish that the effect is plausible or educationally meaningful for the proposed intervention.
For example, a relatively small improvement in achievement may still be important when an educational intervention is inexpensive and scalable.
Conversely, assuming an unrealistically large effect may produce an inadequately small sample.
Researchers should distinguish the minimum effect of interest from the effect anticipated on the basis of previous evidence.
These values should be justified rather than selected merely to produce a convenient recruitment target.
Fourth, Verify Outcome Variability
Sample size calculations involving continuous outcomes often depend on an anticipated standard deviation or another measure of variability.
For example, an experiment comparing mathematics achievement may require an estimate of the variability of scores under the planned assessment.
AI may insert an arbitrary standard deviation or borrow a value from an unrelated study.
However, variability may differ across populations, instruments, educational levels, and instructional settings.
Researchers should use relevant published evidence, existing institutional data, or appropriately interpreted preliminary information.
When uncertainty remains, sensitivity calculations can show how different plausible values affect the required sample.
Fifth, Verify Statistical Power and Significance Level
In conventional hypothesis-testing frameworks, sample size calculations may require a significance level and desired statistical power.
For example, AI may assume a significance level of 0.05 and 80% power.
These are common planning conventions, but they are not universal requirements.
The appropriate values depend on the research purpose, consequences of errors, disciplinary expectations, and analytical framework.
Researchers should also verify whether the calculation uses a one-sided or two-sided test and whether multiple testing considerations affect the design.
A calculation using the wrong testing assumptions may produce an inappropriate sample size even when the effect size is justified.
Sixth, Verify the Sampling and Allocation Design
Sample size requirements depend partly on how observations are selected or assigned.
For example, independently sampled students differ statistically from students recruited through intact classrooms.
Likewise, a randomized experiment assigning participants equally between two groups may have different requirements from one using unequal allocation.
Researchers should verify whether the calculation accounts for the actual sampling and assignment procedures.
AI should not assume independent observations or equal group sizes merely because these conditions simplify the calculation.
Seventh, Verify the Design Effect
When observations are clustered, the variance of an estimator may differ from that under simple random sampling.
A familiar equal-cluster approximation is:
Illustrative Design Effect
DEFF = 1 + (m − 1)ρ
Where:
m = cluster size.
ρ = intraclass correlation coefficient.
However, the expression depends on particular assumptions and does not apply unchanged to every complex sampling design.
Researchers should verify the cluster size, ICC, number of clusters, and suitability of the adjustment.
An arbitrary design effect such as 1.5 should not be accepted without justification.
The calculation of a design effect may require additional methodological review when cluster sizes vary or the analytical structure is complex.
Eighth, Verify Any Finite Population Correction
A finite population correction may be relevant when sampling without replacement from a defined finite population under an appropriate probability sampling design.
However, AI may apply the correction whenever a researcher provides the total population size.
For example, knowing that a university has 5,000 students does not automatically justify a finite population correction for a randomized intervention experiment.
The correction depends on the sampling design, estimator, and intended inference.
Researchers should verify whether the finite population correction is applicable before accepting the reduced sample size it may produce.
Ninth, Verify the Attrition Adjustment
When a study anticipates loss of usable observations, the recruitment target may need to exceed the analytical sample requirement.
Under a simple constant-retention assumption, the adjustment is:
Basic Attrition Adjustment
Nrecruit = Nrequired / (1 − a)
Where a is the anticipated attrition proportion.
For example, if 200 completed observations are required and 20% attrition is anticipated:
200 / 0.80 = 250 participants.
Simply adding 20% would produce 240 participants, which would yield only 192 expected completed observations under the assumed loss rate.
Researchers should verify both the arithmetic and the justification for the expected attrition.
The adjustment also does not eliminate potential bias arising from missing outcomes.
A separate attrition adjustment audit may be necessary for longitudinal or multi-group studies.
Tenth, Verify Arithmetic and Rounding
Even when the formula and assumptions are appropriate, AI may make errors in numerical substitution, unit conversion, or rounding.
For example, a chatbot may substitute 20 instead of 0.20 for a 20% attrition rate.
It may also round an intermediate value prematurely or round the final required sample downward.
Researchers should reproduce the calculation using appropriate statistical software, a validated computational procedure, or transparent mathematical steps.
When a minimum number of participants is required, the final value should ordinarily be rounded upward to the next whole participant.
For multi-group studies, rounding may need to be applied separately to preserve the required allocation and group-specific sample sizes.
Eleventh, Verify the Planned Statistical Analysis
The sample size calculation should correspond to the analysis that will address the primary research question.
For example, a study planning multiple regression should not automatically use a calculation intended for estimating a population proportion.
Similarly, a repeated-measures experiment may require assumptions about within-participant correlation that are absent from a conventional independent-group calculation.
Complex analytical models may require simulation-based sample size planning.
Researchers should confirm that the calculation addresses the relevant parameter or contrast rather than merely using the name of a familiar statistical test.
Twelfth, Verify Sensitivity to Uncertain Assumptions
Many sample size inputs are estimates rather than known facts.
For example, the anticipated effect size, outcome standard deviation, ICC, and attrition rate may be uncertain before data collection.
Researchers should examine how the required sample changes across plausible alternative values.
AI can help organize these scenarios, but the ranges should be justified rather than invented.
Sensitivity analysis makes uncertainty visible and helps researchers determine whether the recruitment plan remains feasible under less favorable assumptions.
A sample size that appears adequate only under optimistic assumptions may warrant reconsideration.
Can Two Correct Calculators Produce Different Sample Sizes?
Yes. Different procedures may use different approximations, statistical distributions, continuity corrections, estimation methods, or design assumptions.
For example, a normal-approximation calculation and an exact statistical procedure may produce slightly different requirements.
Likewise, a power-based calculation may produce a different sample size from a precision-based calculation because the two approaches address different planning objectives.
Researchers should not assume that disagreement between calculators automatically indicates an error.
Instead, they should identify the methods and assumptions underlying each result and determine which corresponds to the intended analysis.
Is Agreement Between ChatGPT and Statistical Software Enough?
Agreement is useful evidence that the same calculation may have been implemented consistently.
However, it does not establish that the selected method or assumptions are appropriate.
For example, ChatGPT and G*Power may produce matching sample sizes when both use the same assumed effect size and statistical test.
If the effect size is unjustified or the test does not correspond to the planned analysis, the agreement does not validate the research design.
Independent verification must therefore include methodological reasoning as well as numerical reproduction.
What Statistical Software Can Be Used for Verification?
Researchers may use appropriate statistical applications such as R, G*Power, Stata, SAS, or specialized design-specific software.
For example, G*Power supports power analyses for several common statistical tests, while R provides functions and packages for a wide range of statistical models and simulation procedures.
However, software suitability depends on the intended calculation.
A complex cluster-randomized or longitudinal design may require specialized methods beyond a basic independent-group power calculator.
Researchers should document the software, version, procedure, and inputs used to reproduce the calculation.
Should the Sample Size Be Verified by a Statistician?
Statistical consultation may be particularly valuable when the study involves complex sampling, clustering, longitudinal measurements, mediation, structural equation modeling, or other analyses requiring specialized assumptions.
However, not every simple calculation requires a formal statistical consultation.
The level of review should reflect the complexity and consequences of the methodological decision.
For example, a conventional descriptive survey calculation may be verified using established formulas and appropriate software when the assumptions are clear.
A multi-site cluster-randomized trial may require more extensive specialist review.
AI assistance does not eliminate the need for relevant methodological expertise.
What Should Be Documented for Reproducibility?
A defensible sample size justification should allow another researcher to understand and reproduce the calculation.
| Documentation element |
What should be recorded |
| Research objective |
Primary estimand, outcome, or hypothesis. |
| Calculation method |
Formula, statistical procedure, or simulation approach. |
| Planning assumptions |
Effect size, variability, power, precision, and other relevant inputs. |
| Sources of assumptions |
Published studies, preliminary data, institutional records, or substantive justification. |
| Design characteristics |
Sampling, allocation, clustering, repeated measures, or other relevant structures. |
| Adjustments |
Finite population correction, design effect, attrition, and other applicable modifications. |
| Computational verification |
Software, version, procedure, and independently reproduced result. |
| Final target |
Required analytical sample and planned recruitment number. |
The documentation should distinguish verified information from planning assumptions and explain important uncertainties.
Researchers should also follow applicable institutional and publisher policies concerning disclosure of substantive AI assistance.
What If the Verified Sample Size Is Larger Than the Available Population?
Sometimes an appropriate calculation produces a requirement that cannot be met under the available recruitment conditions.
For example, a study may require more independent classrooms than the institution can provide.
The appropriate response is not automatically to reduce the sample size until it becomes convenient.
Researchers may need to reconsider the research question, precision objective, effect of interest, recruitment setting, or design.
However, changes to the calculation assumptions must be scientifically justified.
Feasibility constraints should be acknowledged rather than concealed through optimistic inputs.
Can a Verified Sample Size Guarantee Valid Research Findings?
No. Sample size adequacy addresses only part of research quality.
A study may have sufficient statistical power or precision while suffering from selection bias, poor measurement, confounding, implementation problems, or inappropriate analysis.
Similarly, a large sample does not automatically compensate for an invalid research design.
Researchers should therefore evaluate sample size within the broader methodological framework.
A verified calculation supports planning, but it does not independently establish the validity of the eventual findings.
How Can You Conduct a Complete AI-Assisted Sample Size Audit?
1. Confirm the primary research objective. Identify the parameter, comparison, or hypothesis requiring adequate information.
2. Verify the calculation method. Determine whether the formula or computational procedure matches the design and planned analysis.
3. Audit the assumptions. Examine effect sizes, variability, power, precision, and other required inputs.
4. Check the design structure. Confirm sampling, allocation, clustering, repeated measurements, and relevant dependence assumptions.
5. Verify adjustments. Examine finite population corrections, design effects, attrition, and other modifications.
6. Reproduce the arithmetic. Use an appropriate independent computational method and verify rounding.
7. Conduct sensitivity analyses. Examine how plausible alternative assumptions affect the result.
8. Document the final justification. Record the methods, inputs, sources, software, uncertainty, and recruitment target.
This audit should be completed before the final sample size is incorporated into the research protocol.
Watch Out
Do not confuse reproducing an AI-generated number with validating the sample size. A calculator can confirm the arithmetic of an inappropriate formula just as accurately as it can confirm a suitable one.
