03 · What You Need to Know
Why AI-Generated Sample Size Calculations Require Independent Verification
What Does It Mean for a Sample Size Calculation to Be Correct?
A defensible sample size calculation involves several distinct requirements.
First, the calculation must address the correct statistical or methodological objective. Second, the selected formula or computational procedure must be appropriate. Third, the input values must be justified. Finally, the arithmetic and interpretation must be accurate.
For example, a chatbot may correctly calculate the sample size required to estimate a population proportion with a specified margin of error.
However, if the study is designed to compare two instructional interventions, that calculation may not address the primary research question.
Lakens (2022) emphasizes that sample size justification should reflect the study's inferential goals and the rationale for the information required, rather than relying on an unexplained conventional number.
What Types of Errors Can AI Make in Sample Size Calculations?
| Error type |
What happens |
Illustrative consequence |
| Formula selection error |
AI chooses a calculation method intended for another research objective. |
A survey proportion formula is applied to an experimental comparison. |
| Unsupported input |
AI assumes values that the researcher has not provided or justified. |
A medium effect size is inserted without relevant evidence. |
| Arithmetic error |
AI incorrectly substitutes values or evaluates the expression. |
The calculated number differs from the correct result. |
| Design omission |
AI ignores clustering, unequal allocation, repeated measures, or other relevant features. |
The estimate understates the information required. |
| Adjustment error |
AI incorrectly applies attrition or finite population adjustments. |
The recruitment target is too small or unnecessarily large. |
| Interpretation error |
AI presents a conditional calculation as a universal requirement. |
A sample appropriate for one assumption set is described as sufficient for the entire study. |
These errors may occur independently or together. A calculation can contain correct arithmetic while remaining invalid because its assumptions do not match the study.
Why Does Showing a Formula Not Establish Accuracy?
Displaying a formula makes a calculation easier to inspect, but it does not establish that the formula was selected appropriately.
For example, an AI chatbot may use a conventional formula for estimating a population proportion:
Illustrative Formula
n₀ = Z² × p(1 − p) / e²
Where Z represents the relevant normal critical value, p the anticipated population proportion, and e the desired absolute margin of error.
This formula may be appropriate as a starting point for estimating a proportion under specified simple random sampling assumptions.
However, it is not automatically suitable for comparing two means, testing a regression coefficient, estimating an interaction effect, or planning a cluster-randomized trial.
Researchers should therefore ask why the formula applies before examining whether the arithmetic is correct.
Can AI Produce the Right Number for the Wrong Reason?
Yes. Two different calculations may produce similar numerical results even when they address different research objectives.
Suppose AI recommends 200 participants using a conventional survey formula. An independent power analysis for the researcher's experimental comparison might coincidentally produce a requirement close to 200.
The numerical similarity does not validate the original calculation.
The appropriate sample size must follow from the intended outcome, estimand, statistical procedure, and relevant assumptions.
A correct-looking number is not a substitute for a correct justification.
Can AI Invent an Effect Size Without Telling You?
Yes. When researchers omit essential information, AI may supply familiar conventional values rather than identify the missing assumptions.
For example, it may assume Cohen's d = 0.50 for a comparison between two instructional conditions because that value is commonly described as a medium standardized effect.
However, Cohen's conventional benchmarks are not universal estimates of educational intervention effects.
The effect size used in planning should be justified by the minimum effect of interest, relevant previous evidence, or another defensible substantive rationale.
Choosing an unrealistically large effect can produce an inadequately small sample-size requirement.
AI should identify the effect size as an assumption rather than present it as an established characteristic of the study.
Why Is Statistical Power Important?
In conventional hypothesis-testing frameworks, statistical power is the probability of rejecting a specified null hypothesis when the assumed alternative effect is present.
Researchers often consider power targets such as 80% or 90%, but these values should not be treated as universally appropriate.
Power depends on the assumed effect, statistical test, significance level, variability, sample size, and design structure.
AI may correctly explain power while calculating it for a test that differs from the planned analysis.
For example, a calculation based on an independent two-sample t-test may not adequately represent a clustered intervention study or a repeated-measures analysis.
Researchers should ensure that the power calculation corresponds to the actual analytical strategy.
Can AI Misapply the Finite Population Correction?
Yes. A finite population correction may be relevant when sampling without replacement from a defined finite population under an appropriate probability sampling design.
It is not a universal adjustment for every study conducted within an institution of known size.
For example, knowing that a university has 5,000 students does not automatically justify applying a finite population correction to a randomized intervention trial.
The correction depends on the sampling design and the parameter being estimated.
AI should explain why the correction applies rather than include it simply because a population count was provided.
Can AI Overlook Clustered or Repeated Measurements?
Yes. Sample-size calculations must reflect the dependence structure of observations when that structure affects the intended analysis.
For example, students within the same classroom may have correlated outcomes because they share an instructor and learning environment.
A calculation treating all students as independent may underestimate the uncertainty associated with a classroom-level intervention.
Similarly, repeated measurements from the same participant are not equivalent to independent observations from different participants.
AI may overlook these distinctions when the prompt provides only the total number of observations.
Researchers should independently verify any design effect or clustering adjustment used in the calculation.
Can AI Make Errors in Attrition Adjustments?
Yes. A common mistake is to increase the required sample by the expected attrition percentage rather than dividing by the anticipated retention proportion.
Suppose the study requires 200 completed observations and anticipates 20% attrition.
Illustrative Calculation
Incorrect: 200 × 1.20 = 240 participants.
Appropriate simple adjustment: 200 / (1 − 0.20) = 250 participants.
The second calculation reflects the expectation that 80% of recruited participants will provide usable observations.
However, this adjustment assumes that the attrition estimate is appropriate and does not address potential bias arising from missing outcomes.
More complex designs may require additional considerations, including differential attrition between groups.
Researchers should verify the attrition adjustment separately from the initial sample-size requirement.
Can Repeatedly Asking AI Produce a More Reliable Answer?
Repeated prompting may reveal inconsistencies or encourage the system to reconsider its assumptions.
However, receiving the same number several times does not independently establish correctness.
The chatbot may consistently apply the same inappropriate formula or reproduce the same unsupported assumptions.
Likewise, agreement between different AI systems is not equivalent to independent statistical verification.
Researchers should reproduce the calculation using an appropriate statistical procedure rather than rely on consensus among chatbot responses.
What Tools Can Researchers Use to Verify AI Calculations?
Depending on the study, researchers may use statistical software such as R, G*Power, Stata, SAS, or other suitable applications.
For example, G*Power can support power analyses for several common statistical tests, while R provides functions and packages for a broader range of design-specific calculations.
However, software does not automatically establish that the selected model or assumptions are appropriate.
A researcher can obtain an incorrect sample-size recommendation from validated software by entering inappropriate inputs.
Independent verification should therefore examine both the numerical computation and the scientific rationale for the calculation.
Should Researchers Trust Online Sample Size Calculators More Than AI?
Not automatically. Online calculators may implement specific formulas correctly, but their suitability depends on the assumptions and purposes for which they were designed.
For example, a calculator intended for estimating survey proportions may not be appropriate for planning a randomized experiment.
Researchers should identify the underlying formula, confirm the relevant assumptions, and verify that the calculator addresses the intended statistical question.
The important distinction is not simply between AI and non-AI tools. It is between transparent, appropriate, reproducible calculations and unsupported numerical recommendations.
Can AI Estimate Sample Size Without Complete Information?
AI can develop illustrative scenarios when important inputs are uncertain, provided the assumptions are explicitly identified.
For example, it may compare required sample sizes under several plausible effect sizes or anticipated attrition rates.
Such scenarios can help researchers understand the consequences of uncertainty.
However, a scenario should not be presented as the final required sample size until the planning assumptions have been justified.
If essential inputs are missing, AI should ask for them or explain what cannot yet be determined.
Can AI Determine Sample Size Adequacy After Data Collection?
AI can help evaluate the precision of obtained estimates, confidence intervals, and limitations of the realized sample.
However, calculating observed or post hoc power by substituting the observed effect into a conventional power formula generally adds little useful information beyond the observed test result.
Researchers should instead examine the actual estimates, their uncertainty, the study design, and the scientific importance of the findings.
A completed study should not be declared adequate solely because AI calculates a favorable post hoc power value.
What Should Be Independently Verified?
A defensible review should examine the complete calculation rather than only its final number.
1. Verify the research objective. Confirm which parameter, comparison, or hypothesis the calculation addresses.
2. Verify the formula or procedure. Determine whether it matches the outcome, design, and intended analysis.
3. Verify the inputs. Check the justification for effect sizes, variability, power, precision, and other assumptions.
4. Verify the arithmetic. Independently reproduce the result using appropriate software or a transparent computational method.
5. Verify design adjustments. Examine clustering, unequal allocation, finite population corrections, and repeated measurements where applicable.
6. Verify recruitment adjustments. Check anticipated attrition, nonresponse, and the distinction between completed observations and recruited participants.
7. Examine sensitivity. Determine how the result changes under plausible alternative assumptions.
8. Document the justification. Record the method, software, assumptions, and final recruitment target.
The resulting sample-size justification should be understandable and reproducible without relying on the chatbot's authority.
Watch Out
A precise sample-size estimate is not proof of methodological accuracy. If the formula, inputs, or design assumptions are inappropriate, independently confirming the arithmetic will reproduce the wrong answer more accurately.
