How Do You Estimate Sample Size When No Formal Calculation Is Available?

What Should You Do When You Cannot Calculate a Formal Sample Size?

You are planning an exploratory study of university instructors' experiences with a newly introduced generative AI assessment platform. You have identified the target population, developed a questionnaire, and established the principal outcomes. However, there are no reliable previous estimates of outcome variability, and no closely comparable study provides an effect size that you can confidently use.

You still need to determine how many participants to recruit. Should you select a convenient number, use the sample size reported in a similar publication, or recruit as many participants as possible?

Formal sample size calculations are useful when their inputs and statistical objectives can be specified meaningfully. When those conditions are absent, researchers may need another defensible basis for planning recruitment.

The challenge is to distinguish a transparent planning estimate from a statistically established requirement. A number can be practical and justified without being the result of a conventional power calculation, but its limitations must be understood.

Use a Transparent Justification Based on the Study's Purpose and Available Evidence

How to Develop a Defensible Sample Size Estimate Without Conventional Power Inputs

Why Might a Formal Sample Size Calculation Be Unavailable?

A conventional sample size calculation requires a sufficiently specified statistical objective and assumptions about the data-generating process.

For hypothesis testing, these commonly include the primary outcome, planned statistical test, significance level, desired power, expected effect size, and relevant outcome variability.

For estimation, the researcher may instead need a target level of precision and assumptions about the outcome distribution.

These inputs may be unavailable when a phenomenon is new, previous evidence is limited, the measurement instrument has not been evaluated, or the proposed analysis is unusually complex.

However, missing inputs should not automatically be interpreted as evidence that no calculation is possible. Researchers may be able to examine plausible ranges, use conservative assumptions, or undertake simulation.

The first task is to establish which information is genuinely unavailable and whether it can reasonably be obtained.

Does Every Study Need a Formal Power Calculation?

No. The need depends on the research objective and design.

A confirmatory study designed to detect a specified effect will ordinarily require prospective justification of its statistical operating characteristics, often through a power or precision calculation.

By contrast, a feasibility study may primarily investigate recruitment rates, instrument completion, or implementation procedures. Its sample size may be justified according to the feasibility quantities it needs to assess.

An exploratory study may also use a planning target that reflects its descriptive and analytical objectives, although the resulting estimates and tests may remain imprecise.

The distinction is between a sample size justified for the actual study purpose and one chosen without examining what the study needs to accomplish.

The broader question of whether every quantitative study requires a formal sample size calculation should be considered before selecting an alternative approach.

Begin by Identifying the Primary Purpose of the Study

Sample size cannot be justified independently of the question being answered.

For example, a study estimating the proportion of instructors who use AI-generated feedback has different planning requirements from a study comparing adoption rates between institutional sectors.

A feasibility study examining whether instructors can complete a questionnaire within ten minutes has a different objective again.

Researchers should specify whether the main purpose is estimation, hypothesis testing, feasibility assessment, exploratory association analysis, or preliminary instrument evaluation.

That decision determines which assumptions matter and whether a numerical planning method is available.

Without a clearly defined objective, even a mathematically correct sample size formula may answer the wrong question.

Can You Estimate Sample Size Using Desired Precision?

Yes. When the objective is to estimate a population proportion, a precision-based approach may be available even if no prior prevalence estimate exists.

Under simple random sampling, a commonly used planning approximation for a proportion is:

This is a formal precision calculation, not an informal substitute. It is included because researchers who lack previous prevalence estimates may still have sufficient information to calculate a target.

The approximation assumes an appropriate probability sampling design and should not be applied to a convenience sample as though it guaranteed a conventional margin of error.

Finite population size, complex sampling, nonresponse, and other design features may require adjustments.

What If You Cannot Specify a Meaningful Effect Size?

Researchers should not invent an expected effect merely to complete a power calculation.

For example, a study of a newly developed AI feedback intervention may lack credible evidence about the difference in learning outcomes it could produce.

One option is to identify a smallest effect size of interest based on substantive or practical importance rather than predicting the effect that will actually occur.

Another is to examine a range of plausible effects and report the corresponding sample size requirements.

If no meaningful effect threshold can be specified, a feasibility or preliminary estimation objective may be more defensible than presenting the study as a fully powered confirmatory test.

The distinction between an anticipated effect and a substantively important effect should remain explicit.

How Can Sensitivity Analysis Help?

Sensitivity analysis examines how sample size requirements change under alternative assumptions.

Suppose previous studies provide inconsistent estimates of the variability of a digital literacy score.

Rather than selecting one estimate without justification, the researcher may calculate requirements under several plausible standard deviations.

For example, assumptions of standard deviations of 10, 15, and 20 points may produce substantially different requirements for detecting a fixed mean difference.

The analysis can reveal whether the proposed study remains feasible across plausible scenarios.

When the range is broad, researchers should report the uncertainty rather than present one calculated number as definitive.

Sensitivity analysis does not remove uncertainty about the inputs. It makes the consequences of that uncertainty more visible.

Can Previous Studies Provide a Starting Point?

Previous research can provide useful information about outcome distributions, recruitment rates, measurement reliability, and plausible effects.

However, copying the sample size of a published study is not a sufficient justification.

Two studies may use similar topics but differ in their outcomes, designs, statistical tests, population characteristics, and intended precision.

For example, a published survey of 250 teachers may have been designed for descriptive analysis, while a new study intends to compare several institutional groups.

The previous sample size may be informative about feasibility but not about the adequacy of the proposed comparison.

Researchers should extract relevant assumptions and assess their applicability rather than imitate the final participant count.

Can Pilot Data Help Estimate Sample Size?

Yes. Pilot data may inform recruitment feasibility, outcome variability, questionnaire completion, missingness, and measurement procedures.

For example, a small pilot study may reveal that the outcome varies much more widely than originally anticipated.

However, estimates from small pilots can be unstable, particularly when used to predict treatment effects.

Using an imprecise pilot effect estimate as though it were a reliable expected effect can lead to misleading sample size requirements.

Pilot evidence is often more useful for assessing feasibility and nuisance parameters, such as outcome variability, than for establishing a definitive intervention effect.

Researchers should describe the uncertainty surrounding pilot estimates and avoid presenting preliminary findings as conclusive.

When Is a Feasibility-Based Sample Size Justified?

A feasibility-based target may be appropriate when the principal objective concerns whether research procedures can be implemented.

For example, a pilot study may seek to determine whether university instructors can be recruited across departments, whether the questionnaire is understandable, and how frequently participants complete the study.

The recruitment target should be linked to the feasibility information needed.

If the objective is to estimate a recruitment or completion rate, researchers may consider the expected precision of that rate.

If the objective is to identify major procedural difficulties, a smaller purposively chosen set of participants may be appropriate, depending on the design.

Feasibility-based planning is not a license to choose an arbitrary number. The sample should still be sufficient for the decisions the pilot is intended to inform.

How Should You Use Recruitment Capacity?

Recruitment capacity can establish a realistic upper limit on sample size.

Suppose a researcher can access approximately 180 eligible instructors during the data collection period.

That number provides information about feasibility, but it does not establish that 180 participants are statistically adequate.

The researcher should distinguish the number of eligible people who can be approached from the number expected to provide usable data.

Recruitment planning should consider participation rates, eligibility, incomplete responses, and any expected attrition.

Where the feasible sample cannot support the original objective, the appropriate response may be to revise the design, reduce the scope of inference, or treat the study as exploratory.

Feasibility constrains what can be done; it does not determine what is scientifically sufficient.

Why Is Recruiting as Many Participants as Possible Not Always Enough?

Recruiting the maximum available number may be reasonable in some settings, but it does not by itself establish adequacy.

A sample of 500 volunteers may be unsuitable for estimating population prevalence if recruitment is strongly selective.

Similarly, 100 observations may be inadequate for an analysis involving rare outcomes or numerous model parameters.

Researchers should distinguish sample size from sampling quality, measurement quality, and analytical complexity.

More participants can improve statistical precision under appropriate assumptions, but cannot automatically correct systematic selection bias or design weaknesses.

The recruitment target should therefore be evaluated in relation to the intended analysis and inference.

Can Simulation Be Used When Conventional Formulas Are Inadequate?

Yes. Simulation can be useful when the study design or planned analysis is too complex for a convenient closed-form calculation.

Researchers can generate hypothetical datasets under specified assumptions, apply the intended analysis, and examine performance across candidate sample sizes.

For example, a study involving repeated observations nested within instructors and institutions may require assumptions about within-person and within-institution correlations.

Simulation can evaluate statistical power, estimator bias, confidence interval coverage, or estimation precision under alternative scenarios.

However, simulation is not assumption-free. Its results depend on the data-generating mechanisms specified by the researcher.

When those mechanisms are uncertain, multiple plausible scenarios should be examined.

Can a Rule of Thumb Ever Be Defensible?

A rule of thumb may provide a preliminary planning reference, but it should not be treated as a universal statistical requirement.

For example, claims that every survey requires at least 100 respondents or that every regression requires a fixed number of observations per predictor ignore important differences in outcomes, effect sizes, distributions, and analytical goals.

Some methodological recommendations apply to narrowly defined situations and may be useful when their assumptions match the proposed study.

However, a numerical convention should be identified as a heuristic and its limitations explained.

Researchers should avoid converting an approximate recommendation into an apparently authoritative minimum.

What If the Study Uses Non-Probability Sampling?

Sample size planning remains important in non-probability studies, but conventional probability-sampling precision claims may not apply.

For example, a voluntary online survey of instructors may obtain 400 responses, yet the sample may systematically overrepresent educators interested in generative AI.

A conventional sample size formula cannot eliminate this selection mechanism.

Researchers may still plan sufficient observations for exploratory comparisons, descriptive summaries, or model-based analyses under explicit assumptions.

However, they should avoid claiming that the achieved sample has a probability-based margin of error merely because it meets a numerical target.

The relationship between sample size and inference depends on both the number of observations and the selection process.

How Do You Account for Nonresponse and Missing Data?

The intended number of completed, usable observations is different from the number of people who must be invited.

Suppose a study requires 150 completed questionnaires and anticipates a 60% completion rate among eligible invitees.

The corresponding invitation target would be approximately 150 / 0.60 = 250 eligible invitations.

However, this adjustment assumes the anticipated rate is applicable and does not resolve nonresponse bias.

Researchers should distinguish refusal, ineligibility, incomplete questionnaires, and longitudinal attrition because they affect recruitment and analysis differently.

For detailed planning, see adjusting sample size for nonresponse and adjusting sample size for dropout or attrition.

How Should an Informal Sample Size Estimate Be Reported?

The methodology should state the primary objective, the basis for the recruitment target, the information available at planning, and the assumptions or constraints considered.

If the target was based on feasibility, explain the recruitment setting and why the number was sufficient for the feasibility questions.

If plausible parameter ranges were examined, describe those scenarios and their implications.

If a pilot provided preliminary estimates, explain their uncertainty.

Researchers should distinguish an a priori planning justification from retrospective arguments based on the achieved sample.

A transparent explanation of limited information is preferable to a precise-looking calculation based on invented inputs.

Planning an Exploratory Survey Without Reliable Preliminary Estimates

The researcher plans to invite all 300 eligible instructors available through the approved recruitment channels, with an anticipated 180 usable responses.

This number is not described as a statistically calculated requirement. It is an expected recruitment outcome under a stated assumption.

Suppose the final study obtains 165 usable questionnaires. The researcher reports the achieved number, evaluates feasibility indicators, and presents descriptive results with appropriate caution.

If the study later proceeds to a confirmatory comparison of AI assessment outcomes, the new design will require its own sample size justification using a defined primary outcome, relevant effect or precision target, and suitable statistical assumptions.

The initial exploratory study is defensible when its conclusions remain aligned with its feasibility and descriptive objectives.

Common Mistakes When Estimating Sample Size Without Formal Calculations

How to Choose a Defensible Alternative When Formal Inputs Are Missing

Begin by establishing whether the study genuinely lacks the information required for a conventional calculation or whether plausible assumptions can be developed.

In the methodology section, distinguish the basis for choosing the recruitment target from the evidence supporting the eventual analysis.

Do not present an arbitrary number as a calculated minimum, and do not claim that feasibility alone establishes statistical adequacy.

A defensible estimate makes its assumptions visible and keeps the research conclusions within the limits of the evidence the planned sample can provide.

Before Finalizing an Estimated Sample Size, Verify the Justification

Frequently Asked Questions About Sample Size Without Formal Calculations

A Defensible Estimate Requires Reasoning, Even Without a Formal Formula

How to Cite This Guide

This guide is intended to be read, shared, and used in research, teaching, and academic work. If you draw on its ideas, explanations, or other content, please acknowledge the source by citing the guide. Doing so gives appropriate credit and helps your readers locate the original resource.