03 · What You Need to Know
How to Examine the Assumptions Behind AI-Generated Sample Size Calculations
What Are Sample Size Assumptions?
Sample size assumptions are the statistical, methodological, and substantive conditions used to determine how much information a study requires.
Some assumptions specify numerical inputs, such as the anticipated standard deviation of an outcome or the effect size the study aims to detect.
Others concern the structure of the research, including whether observations are independent, participants are randomly sampled, groups have equal allocation, or repeated measurements are correlated.
These assumptions influence the calculation because they determine how sample size relates to statistical uncertainty or power.
Lakens (2022) emphasizes the importance of transparent sample size justification, including the assumptions and research objectives underlying the selected approach.
AI can help make these assumptions visible, but identifying an assumption is not equivalent to demonstrating that it holds.
Which Assumptions Are Commonly Required?
| Assumption or input |
Why it matters |
What requires justification |
| Primary estimand or hypothesis |
Determines what the calculation must estimate or test. |
The relationship between the research question and intended analysis. |
| Effect size |
Influences the sample required to detect a specified difference or relationship. |
The scientific importance and plausibility of the target effect. |
| Outcome variability |
Affects the uncertainty of estimated means, differences, or other parameters. |
The anticipated standard deviation or relevant variance structure. |
| Statistical power |
Specifies the probability of detecting the assumed effect under the planned test. |
The selected power target and testing framework. |
| Significance level |
Determines the Type I error criterion for conventional hypothesis testing. |
The chosen threshold and whether multiple testing requires consideration. |
| Target precision |
Specifies the desired uncertainty of an estimate. |
The acceptable confidence interval width or margin of error. |
| Allocation and sampling design |
Determines how observations are selected or distributed across groups. |
Unequal allocation, clustering, stratification, and other design features. |
| Missing data or attrition |
Affects the number of usable observations and potentially the validity of inference. |
The anticipated loss rate and assumptions about missingness. |
Not every calculation requires all these inputs. For example, estimating a single population proportion with a desired margin of error does not ordinarily require specifying statistical power.
AI should identify assumptions relevant to the selected calculation rather than provide the same generic list for every research design.
Why Must the Primary Research Objective Be Specified First?
The research objective determines which quantity requires adequate information.
For example, estimating the prevalence of generative AI use among university students is different from detecting an achievement difference between two instructional conditions.
The first may involve an anticipated population proportion and desired margin of error. The second may require a target mean difference, outcome variability, statistical power, and allocation ratio.
If AI assumes the wrong objective, it may identify assumptions appropriate to a different calculation.
Researchers should therefore verify the sample size calculation method before accepting its associated inputs.
Why Is the Assumed Effect Size So Important?
In power-based calculations, the target effect is often one of the most influential inputs.
Smaller effects generally require larger samples to detect with a specified level of power, holding other relevant conditions constant.
For example, detecting a small improvement in mathematics achievement may require substantially more participants than detecting a large improvement.
AI may recommend Cohen's conventional small, medium, or large effect size categories without considering whether those values are appropriate for the intervention.
However, Cohen (1988) did not establish universal effect sizes for every educational intervention or research context.
Researchers should distinguish between an effect considered practically meaningful and an effect considered plausible based on prior evidence.
These values may differ, and the distinction should inform sample size planning.
What Is the Minimum Effect of Interest?
The minimum effect of interest is the smallest effect that the researcher considers sufficiently meaningful for the study's scientific or practical purpose.
For example, a modest improvement in mathematics achievement may be important if an intervention is inexpensive and can be implemented widely.
Conversely, the same improvement may be insufficient to justify an intervention requiring substantial resources.
The minimum effect of interest should therefore be informed by substantive considerations rather than statistical conventions alone.
AI can help researchers articulate the decision context, but it cannot independently determine which educational improvement is worth detecting.
That judgment may require relevant literature, stakeholder input, and consideration of the consequences of the intervention.
Where Should Outcome Variability Assumptions Come From?
For continuous outcomes, the anticipated standard deviation can substantially influence the required sample size.
Researchers may obtain planning estimates from previous studies using comparable measures and populations, existing institutional data, or appropriately interpreted preliminary evidence.
However, variability may differ across populations, assessment instruments, and educational settings.
For example, a standard deviation reported for a mathematics achievement test in secondary schools may not be appropriate for a university-level assessment with a different scoring scale.
AI should not invent a standard deviation or treat an unrelated published value as directly transferable.
When uncertainty remains, researchers may examine plausible ranges and evaluate how the sample size changes.
Are 80% Power and a 5% Significance Level Mandatory?
No. These values are common conventions in many research settings, but they are not universal methodological requirements.
Statistical power reflects the probability of rejecting the null hypothesis when a specified alternative effect is present under the assumed model.
The significance level specifies the probability of a Type I error under the relevant null hypothesis and testing assumptions.
Researchers should choose these values according to the research purpose, consequences of errors, disciplinary expectations, and analytical framework.
AI may use conventional values for an illustrative calculation, but it should identify them explicitly as planning choices rather than established facts about the study.
What Assumptions Matter for Precision-Based Calculations?
Precision-based calculations focus on the desired uncertainty of an estimate rather than the probability of rejecting a hypothesis.
For example, estimating the proportion of university students who use generative AI may require assumptions about the anticipated proportion, confidence level, margin of error, and sampling design.
For a conventional simple random sample, the anticipated proportion affects the variance through p(1 − p).
When the true proportion is unknown, p = 0.50 is often used because it maximizes this variance term.
However, this is a conservative planning choice under the relevant formula, not an empirical claim that half the students use AI.
AI should distinguish the assumed value from an observed population characteristic.
Why Does Independence Matter?
Many conventional sample size formulas assume independent observations.
However, participants recruited from the same classroom, school, or institution may have correlated outcomes.
For example, students within one classroom may share instructional experiences and assessment conditions.
If a calculation treats these observations as independent when they are substantially correlated, it may overstate the information available from the sample.
Researchers should examine the sampling and assignment structure and use design-specific calculations when necessary.
AI should not infer independence simply because the dataset contains separate rows for individual participants.
What Is the Role of the Intraclass Correlation Coefficient?
The intraclass correlation coefficient, commonly abbreviated ICC, describes the similarity of observations within clusters under a specified model.
In cluster sampling or cluster-randomized studies, the ICC can influence the effective information provided by the sample.
For equal-sized clusters, a familiar approximation for the design effect is:
Illustrative Design Effect
DEFF = 1 + (m − 1)ρ
Where:
m = number of observations per cluster.
ρ = intraclass correlation coefficient.
This expression depends on particular assumptions and does not apply unchanged to every clustered design.
AI may recommend an arbitrary ICC, such as 0.05, without evidence that it is appropriate for the population and outcome.
Researchers should use relevant empirical estimates or examine sensitivity across plausible values.
The calculation of a design effect should be verified separately.
Why Does Allocation Ratio Affect Sample Size?
When a study compares two or more groups, the distribution of participants across conditions affects statistical precision.
Equal allocation is often efficient when group variances and observation costs are comparable.
However, some studies may use unequal allocation for ethical, operational, or other justified reasons.
A calculation assuming equal allocation may underestimate the total sample needed when the actual design uses a substantially unequal ratio.
AI should therefore request the intended allocation ratio rather than automatically assume that groups will contain equal numbers.
What Assumptions Are Required for Repeated-Measures Studies?
Repeated-measures calculations may depend on the number and timing of measurements, within-participant correlations, outcome variability, and the planned analytical model.
For example, a pretest-posttest comparison may have different requirements from a longitudinal study examining changes across several semesters.
Repeated observations may improve precision when their correlation structure is appropriately modeled, but they do not automatically substitute for recruiting additional independent participants.
AI should identify the intended longitudinal contrast and covariance assumptions before recommending a sample size.
How Should Attrition Assumptions Be Evaluated?
Attrition assumptions estimate how many recruited participants may fail to provide usable observations at the required follow-up or analysis stage.
For example, a study requiring 200 completed observations and anticipating 20% attrition may use a simple recruitment adjustment:
Illustrative Attrition Adjustment
Required recruitment = 200 / (1 − 0.20) = 250
However, the assumed 20% loss should be justified using relevant prior studies, institutional experience, or other defensible evidence.
Attrition may also differ between intervention groups or be related to outcomes.
Increasing the recruitment target does not eliminate missing-data bias.
Researchers should distinguish the arithmetic of attrition adjustment from the assumptions required for valid analysis of incomplete data.
Can AI Identify Assumptions for Regression and Structural Equation Modeling?
AI can help identify relevant model-specific requirements, but these analyses do not have one universal sample size assumption set.
For regression, planning may depend on the primary coefficient or hypothesis, anticipated effect, predictor relationships, residual variability, and estimation method.
For structural equation modeling, assumptions may involve factor loadings, parameter values, model identification, estimation, missing data, and the performance criteria being evaluated.
Wolf and colleagues (2013) demonstrate that sample size requirements for structural equation models vary with model characteristics.
Researchers should not rely on universal rules such as ten respondents per item without evaluating the actual model.
Simulation-based planning may be useful for complex models, provided the simulated conditions are justified.
Can AI Distinguish Assumptions From Known Facts?
AI can be instructed to label inputs according to their evidentiary status.
For example, the number of students enrolled at a university may be verified from institutional records, while the expected intervention effect may remain uncertain.
Similarly, a researcher may know the planned allocation ratio but have only preliminary information about attrition.
Separating known conditions from assumptions helps prevent hypothetical values from being presented as established evidence.
| Input status |
Meaning |
Illustrative example |
| Verified |
Supported by an appropriate source or established protocol decision. |
Confirmed number of eligible classrooms. |
| Evidence-informed estimate |
Based on relevant empirical information with acknowledged uncertainty. |
Expected achievement-score standard deviation from comparable studies. |
| Planning choice |
Selected according to the research objective and inferential requirements. |
Target power or desired confidence interval width. |
| Hypothetical assumption |
Used to explore a scenario without sufficient evidence for a final estimate. |
An illustrative attrition rate used in sensitivity analysis. |
These distinctions should remain visible in the final sample size justification.
What If the Required Assumptions Are Unknown?
Missing assumptions should not automatically be replaced with conventional values.
Researchers may consult relevant literature, examine available administrative data, or conduct preliminary investigations when appropriate.
They may also perform sensitivity analyses using plausible ranges.
For example, if the anticipated intervention effect is uncertain, calculating sample sizes across several defensible effect sizes can reveal how strongly the requirement depends on that assumption.
However, sensitivity analysis does not make arbitrary inputs scientifically valid.
Its value lies in showing how conclusions change under explicitly stated alternatives.
Can AI Verify That the Assumptions Are Correct?
Not independently. AI can compare supplied assumptions with methodological principles and identify apparent inconsistencies.
However, it cannot establish that the anticipated effect, standard deviation, ICC, or attrition rate accurately represents the future study population without appropriate evidence.
Even assumptions based on previous studies may not transfer perfectly to a new context.
Researchers should therefore document the sources, limitations, and sensitivity of the planning values.
Independent verification should examine both the assumptions and the resulting calculations.
How Can You Conduct an AI-Assisted Assumption Audit?
1. Define the primary statistical objective. Identify the estimand, hypothesis, or precision requirement.
2. Confirm the calculation method. Establish that the formula or computational procedure matches the design.
3. List every required input. Include effect sizes, variability, power, precision, allocation, dependence, and attrition as applicable.
4. Classify the evidence for each input. Distinguish verified values, empirical estimates, planning choices, and hypothetical scenarios.
5. Examine the research context. Determine whether prior estimates are relevant to the intended population and measurements.
6. Conduct sensitivity analyses. Explore how plausible alternative values affect the required sample size.
7. Independently verify the calculation. Use appropriate statistical software and methodological references.
8. Document unresolved uncertainty. Explain which assumptions remain uncertain and how they affect the final recruitment target.
This process supports transparent planning without treating AI-generated assumptions as empirical evidence.
Watch Out
Do not allow AI to fill missing sample size inputs silently. A conventional value may be convenient for illustrating a calculation, but it becomes a defensible planning assumption only when its selection is justified for the research question and context.
