Does Every Quantitative Study Need a Formal Sample Size Calculation?

Must You Calculate Sample Size Formally Before Conducting Quantitative Research?

You are preparing a quantitative study examining university instructors' experiences with generative AI assessment tools. Your research adviser asks for a sample size calculation, but you are uncertain which formula applies.

The study is primarily descriptive. You intend to report patterns of AI use, perceived implementation barriers, and questionnaire completion. You are not testing the effectiveness of an intervention against a control group.

Does the study still require a statistical power calculation? Could you justify the sample size using the number of eligible instructors, the precision of descriptive estimates, or the feasibility of recruitment?

Researchers sometimes treat formal sample size calculations as mandatory components of every quantitative methodology. Others assume that calculations are unnecessary whenever a study is exploratory or uses convenience sampling.

Neither position adequately distinguishes the different purposes of quantitative research. The important question is what evidence the study intends to produce and which planning procedure can establish whether the proposed sample is sufficient for that purpose.

Not Every Quantitative Study Needs a Power Calculation, but Every Sample Size Needs Justification

When Formal Sample Size Calculations Are Necessary and When Alternatives Are Defensible

What Counts as a Formal Sample Size Calculation?

A formal sample size calculation is a quantitative planning procedure that relates the required number of observations to a defined statistical objective and explicit assumptions.

For hypothesis testing, the calculation commonly specifies an effect of interest, significance level, statistical power, outcome variability, and planned statistical test.

For estimation, it may specify a target confidence interval width or margin of error.

More complex designs may use simulation to evaluate statistical operating characteristics across candidate sample sizes.

These procedures are not interchangeable. A power calculation evaluates the probability of detecting a specified effect under assumed conditions, while a precision calculation evaluates the expected uncertainty of an estimate.

Researchers should therefore identify the statistical objective before choosing a formula or software procedure.

What Is the Difference Between a Sample Size Calculation and a Sample Size Justification?

A calculation produces a numerical requirement under stated assumptions. A justification explains why the proposed number of observations is appropriate for the study.

A formal calculation is often an important part of a justification, but it does not constitute the entire justification.

For example, a calculated requirement of 300 completed observations is not sufficient if the sampling frame systematically excludes the population group central to the research question.

Conversely, a feasibility study may provide a defensible justification without calculating power for an intervention effect that the study is not designed to test.

Lakens' work on sample size justification provides a useful methodological account of different approaches to explaining sample size decisions.

When Is a Formal Power Calculation Particularly Important?

Prospective power calculations are particularly important when the primary purpose is to test a specified hypothesis and the research design permits meaningful assumptions about the effect and outcome distribution.

For example, a randomized study comparing AI-generated feedback with conventional instructor feedback may aim to detect a minimum meaningful difference in student writing performance.

The researchers would ordinarily specify the primary outcome, meaningful difference, expected variability, significance level, desired power, and allocation ratio.

If students are assigned by class rather than individually, clustering must also be considered.

Without such planning, the study may recruit too few participants to detect a meaningful effect reliably, or substantially more participants than necessary for its stated objective.

The importance of a calculation increases when the study is confirmatory, resource-intensive, or intended to support consequential decisions.

Do Descriptive Quantitative Studies Need Power Analysis?

Not necessarily. A descriptive study may aim to estimate a population mean, proportion, or other parameter rather than test a specified effect.

For example, a researcher may want to estimate the percentage of university instructors who use generative AI for formative assessment.

In that situation, a precision-based calculation may be more relevant than a conventional power calculation.

The researcher could specify the desired margin of error, confidence level, and anticipated prevalence under an appropriate probability sampling design.

However, a descriptive study is not automatically exempt from numerical planning. If precise population estimation is its central purpose, a formal precision calculation may be important.

The appropriate method follows the intended inference rather than the label descriptive.

How Does Precision-Based Sample Size Planning Work?

For a population proportion estimated using simple random sampling, a common initial planning approximation is:

The choice p = 0.50 maximizes the variance term for a binary proportion and provides a conservative initial assumption under this formula.

Finite population corrections, complex sampling, and nonresponse may change the planning requirement.

Importantly, this formula does not provide a valid probability-based margin of error for an unrestricted convenience sample merely because the numerical target has been achieved.

What About Pilot and Feasibility Studies?

Pilot and feasibility studies frequently have objectives that differ from those of definitive confirmatory research.

For example, an external pilot of an AI-supported learning intervention may investigate whether eligible students can be recruited, whether participants complete the intervention, and whether outcome measurement procedures function adequately.

The sample size should be justified according to these feasibility objectives.

Where the study estimates a recruitment or retention rate, the expected precision of that estimate may inform the target.

Where the purpose is procedural evaluation, practical considerations and the range of conditions requiring assessment may be important.

The CONSORT extension for randomized pilot and feasibility trials emphasizes reporting the rationale for sample size.

A pilot should not be presented as adequately powered to establish treatment effectiveness merely because it includes a conventional number of participants.

Do Exploratory Studies Require a Formal Calculation?

Exploratory studies may not have a single prespecified hypothesis or effect size suitable for conventional power analysis.

For example, a researcher may investigate preliminary associations among AI literacy, perceived usefulness, and reported technology practices.

The study may reasonably be framed as exploratory when the principal aim is to characterize patterns or develop hypotheses.

Nevertheless, the sample size should be evaluated in relation to the planned analyses, expected variability, missing data, and limitations of the recruitment design.

Exploratory does not mean that any participant count is scientifically adequate.

Researchers should also avoid presenting exploratory findings as though they resulted from a fully powered confirmatory design.

What If the Study Uses a Census?

A census attempts to include every eligible unit in a defined population.

For example, a researcher may invite all 85 instructors employed in a particular academic department.

If the objective concerns that finite department population, the researcher may not need to calculate how many instructors to select because the intended design includes everyone.

However, a census invitation does not guarantee complete participation.

If only 55 instructors respond, nonresponse may affect the findings. The researcher must distinguish the intended census from the achieved respondent dataset.

Furthermore, analyzing all members of a finite institution does not automatically justify generalization to other institutions or a broader hypothetical population.

The census rationale should therefore identify the target population, coverage, participation, and inferential limits.

What If You Are Using an Existing Dataset?

Secondary-data research often begins with a fixed number of available observations.

For example, an institutional dataset may contain records for 2,400 students who used a learning platform during a previous academic year.

The researcher cannot retrospectively determine how many observations were originally collected for the new research question.

Instead, the methodology should explain the dataset's origin, eligibility restrictions, available sample, missing data, and information relevant to the intended analysis.

Precision assessments, simulation, or evaluation of detectable effect sizes may help characterize what the available data can support.

However, the achieved sample should not be presented as though it resulted from a prospective calculation undertaken before the data existed.

Does Convenience Sampling Eliminate the Need for Sample Size Planning?

No. Convenience sampling affects the basis for population inference, but it does not make sample size irrelevant.

For example, a voluntary online survey may need sufficient observations to examine exploratory associations or estimate model parameters with reasonable stability.

However, a conventional probability-sampling margin of error cannot be justified solely through sample size when inclusion probabilities are unknown.

Researchers should distinguish statistical precision conditional on modeling assumptions from population representativeness.

A large convenience sample may remain systematically biased, while a small probability sample may be more defensible for certain population estimates.

The sampling mechanism and sample size should be evaluated together.

What If the Study Uses Regression or Predictive Modeling?

Regression and predictive modeling require attention to analytical complexity, outcome information, and estimation stability.

For example, a study predicting students' AI-related academic misconduct concerns may include multiple predictors and interaction terms.

A fixed rule such as ten participants per predictor may be insufficient because the required information depends on the outcome distribution, model complexity, anticipated predictive performance, and other factors.

For binary outcomes, the number of observed events can be particularly consequential.

Specialized sample size methods for prediction model development, including those developed by Riley and colleagues, may be more appropriate than a generic hypothesis-testing power calculation.

Researchers should select planning methods suited to the statistical model rather than assume that all quantitative analyses require the same formula.

What If No Reliable Effect Size or Variability Estimate Exists?

Missing planning information does not justify inventing assumptions.

Researchers may examine plausible parameter ranges, use relevant previous evidence, conduct preliminary pilot work, or specify a smallest effect of substantive interest.

For complex designs, simulation may help evaluate candidate sample sizes under alternative scenarios.

If the uncertainty is too substantial to support a confirmatory design, the researcher may need to revise the objective or undertake a preliminary study.

The approaches discussed in estimating sample size when no formal calculation is available can help distinguish defensible alternatives from arbitrary participant targets.

Can Feasibility Alone Justify the Sample Size?

Feasibility can provide a legitimate constraint and may be central to the purpose of a feasibility study.

However, feasibility does not establish that a sample is sufficient for every intended statistical inference.

Suppose a researcher can recruit only 60 instructors but wants to detect a small difference between two intervention groups.

The fact that 60 participants are available does not establish adequate power for that difference.

The researcher may need to revise the research question, select a different outcome, use a more efficient design, or characterize the investigation as preliminary.

Feasibility should be reported as a constraint rather than converted into an unsupported statistical requirement.

What Is Wrong with Universal Sample Size Rules?

Rules such as every quantitative study needs at least 100 participants or every survey needs 384 respondents overlook the assumptions underlying sample size planning.

The number 384, for example, is closely associated with a particular proportion-estimation scenario using a 95% confidence level, a five-percentage-point margin of error, and a conservative prevalence assumption.

It is not a universal requirement for experimental, correlational, predictive, or qualitative research.

Similarly, copying a sample size from a published study does not establish adequacy when the research objectives or statistical models differ.

Numerical conventions may sometimes provide preliminary reference points, but they should not replace design-specific reasoning.

Should You Calculate Power After Collecting the Data?

Post hoc power calculated by substituting the observed effect into a conventional power formula is generally not a useful measure of the evidential strength of the completed study.

Such calculations are closely tied to the observed test statistic and typically add little information beyond the result already obtained.

After data collection, researchers should report the achieved sample, effect estimates, uncertainty intervals, missing data, and relevant limitations.

Prospective power analysis addresses planning before the study. It should not be reconstructed retrospectively as though it justified the original recruitment decision.

Where the sample was fixed in advance, a sensitivity analysis using substantively meaningful effect sizes may help explain the design's limitations, provided it is not confused with observed post hoc power.

How Should Sample Size Be Justified in the Methodology?

The methodology should identify the primary research objective and explain why the chosen sample size is appropriate for it.

For a power calculation, report the statistical test or model, effect size assumption, significance level, desired power, allocation, and relevant design adjustments.

For a precision calculation, report the target parameter, confidence level, anticipated variability, and desired interval width or margin of error.

For feasibility-based planning, explain the procedural objectives, recruitment capacity, and information the sample is expected to provide.

For a census or secondary-data study, explain the defined population or dataset and the limitations of the available observations.

In every case, distinguish the planned sample from the achieved sample and avoid overstating what the available evidence can support.

Three Quantitative Studies That Need Different Sample Size Justifications

For the descriptive survey, the researchers specify a 95% confidence level and a five-percentage-point margin of error. Assuming simple random sampling and an unknown prevalence, the conservative initial requirement is approximately 385 completed observations.

For the randomized intervention, the researchers cannot determine the required number without specifying the meaningful writing-score difference, expected standard deviation, power, significance level, and other design characteristics.

For the feasibility study, the researchers may recruit a smaller number of instructors to evaluate whether the questionnaire and recruitment procedures function as intended. The target should be linked to those feasibility objectives, not to a claim of intervention effectiveness.

All three studies require sample size justification. However, applying the same conventional power formula to all three would be methodologically inappropriate.

The appropriate justification follows the evidence each study is intended to produce.

Common Misunderstandings About Formal Sample Size Requirements

How to Decide Which Sample Size Justification Your Study Needs

Begin by identifying the primary purpose of the quantitative investigation rather than choosing a formula based only on the research design label.

In the methodology section, explicitly identify the basis of the sample size justification and distinguish it from the achieved number of participants.

Do not claim that every quantitative study requires the same calculation. Equally, do not interpret the absence of a conventional power calculation as permission to recruit an arbitrary number.

The defensible choice is the planning approach that addresses the study's actual statistical or procedural objective.

Before Deciding Whether You Need a Formal Calculation, Check the Study Objective

Frequently Asked Questions About Formal Sample Size Calculations

Formal Calculations Depend on Purpose, but Sample Size Justification Is Always Important

How to Cite This Guide

This guide is intended to be read, shared, and used in research, teaching, and academic work. If you draw on its ideas, explanations, or other content, please acknowledge the source by citing the guide. Doing so gives appropriate credit and helps your readers locate the original resource.