Should Sample-Size Sensitivity Analysis Be Part of Study Planning?

Is One Sample-Size Calculation Enough to Justify a Study?

A researcher calculates that 120 participants are required to achieve 80% statistical power. The result appears precise, and the number fits the available recruitment budget.

But the calculation assumes a particular effect size, outcome standard deviation, and participant retention rate. What happens if the true effect is smaller, the outcome more variable, or attrition higher than anticipated?

These possibilities are not unusual. Sample-size calculations necessarily rely on assumptions made before the relevant population parameters are known. A sensitivity analysis helps researchers determine whether their proposed design is reasonably informative across credible alternatives or depends heavily on one uncertain estimate.

Sensitivity Analysis Is Often Worth Including When Planning Assumptions Are Uncertain

How Sample-Size Sensitivity Analysis Strengthens Study Planning

What Is Sample-Size Sensitivity Analysis?

Sample-size sensitivity analysis examines how the results of a sample-size or power calculation change when selected assumptions are varied.

Instead of calculating the required sample size under one anticipated effect, for example, researchers may repeat the calculation under several plausible effects. They may also investigate alternative standard deviations, attrition rates, intracluster correlations, or other parameters relevant to the study design.

The analysis can be conducted in either direction. Researchers may calculate the sample size required under different assumptions or hold a proposed sample size fixed and examine the statistical power or precision it would provide.

The latter is especially useful when recruitment resources are constrained and the researcher needs to understand the consequences of a feasible sample.

Sensitivity analysis should be distinguished from changing assumptions until a preferred sample size appears. Its purpose is to expose uncertainty, not to justify a predetermined number.

Why Is Sensitivity Analysis Important Before Data Collection?

Most sample-size calculations are conditional on quantities that are not known exactly when the study is designed.

For example, previous research may suggest an intervention effect of d = 0.40, but the estimate may have substantial uncertainty. If the true effect is d = 0.25, a study powered only for d = 0.40 may have much less power than intended.

The initial calculation is not mathematically incorrect. It describes the design's operating characteristics under the specified effect. The difficulty arises when researchers interpret those characteristics as though they apply regardless of the true effect.

Lakens (2022) emphasizes that sample-size justification should be connected to the inferential goals of a study. Sensitivity analysis can support that justification by showing how the proposed design performs under alternative assumptions.

Which Assumptions Should Be Tested?

The most useful sensitivity analysis focuses on parameters that are uncertain and have meaningful consequences for the study's operating characteristics.

Not all parameters need to be varied. A researcher should prioritize those that materially influence the design and cannot be specified confidently.

For example, uncertainty about outcome variability may deserve more attention than small differences in a well-established administrative recruitment rate.

How Do You Choose the Values to Examine?

Alternative values should reflect credible uncertainty rather than arbitrary numerical increments.

Relevant evidence may come from previous studies, systematic reviews, pilot data, established measurement properties, or substantive judgments about the effects worth detecting.

For example, if comparable studies suggest standardized effects between 0.25 and 0.45, those values may provide a reasonable starting point for evaluating alternative power calculations.

However, the observed range in previous studies is not necessarily the plausible range for the proposed population. Researchers should consider sampling error, differences in design, and selective reporting.

The distinction between the expected effect and the smallest meaningful effect is particularly relevant. The most likely effect and the effect a study must be capable of detecting need not be identical.

Where a defensible smallest effect size of interest exists, it can provide a substantive reference point for evaluating whether the proposed design is sufficiently sensitive.

What Is the Difference Between Required-Sample-Size Analysis and Fixed-Sample Power Analysis?

Both approaches examine design sensitivity, but they answer different questions.

The first approach is useful when recruitment targets remain flexible. The second is useful when resources impose a practical upper limit on sample size.

Both are prospective planning analyses. Neither should be confused with calculating so-called observed or post hoc power by substituting the effect estimated from completed study data into a conventional power formula.

Observed power calculated from the realized effect typically adds little information beyond the corresponding test statistic and p-value. Interpreting the completed study's effect estimate and confidence interval is generally more informative.

How Does Effect Size Influence a Sensitivity Analysis?

For a conventional two-sided comparison of two independent means with equal group sizes and a common population standard deviation, the normal-approximation sample-size formula is:

The results show that relatively modest differences in the assumed effect can substantially alter the required sample size.

Importantly, the analysis does not determine which effect is true. It reveals the consequences of alternative assumptions.

What Is a Power Curve, and When Is It Useful?

A power curve displays the statistical power of a specified design across different assumed population effects or sample sizes.

For a fixed sample, power generally increases as the magnitude of the true effect moves farther from the null value under a conventional superiority test.

A power curve can help researchers identify the range of effects for which the proposed design is relatively informative.

For example, a study might have approximately 80% power at d = 0.40, substantially less power at d = 0.25, and greater power at d = 0.55.

Such a curve is often more informative than reporting a single power value because it illustrates the design's performance across the effects that researchers may reasonably encounter.

However, a power curve does not indicate the probability that any particular effect is true. It describes conditional operating characteristics.

Should You Vary Multiple Parameters Together?

Yes, when the study's performance depends materially on several uncertain inputs.

For instance, an educational trial may have uncertainty about both the intervention effect and the intracluster correlation among students within classes.

Evaluating only effect size while assuming a fixed intracluster correlation could overlook an important source of uncertainty.

Researchers may therefore construct a scenario matrix in which each row represents a credible combination of assumptions.

However, care is required when parameters are mathematically or empirically related. If standardized effect size is calculated as a raw difference divided by the standard deviation, varying all three independently may produce inconsistent scenarios.

The process of constructing plausible joint assumptions should account for such relationships.

Does Sensitivity Analysis Have to Use Simulation?

No. Many conventional sample-size sensitivity analyses can be performed by repeating an analytical calculation across a selected range of inputs.

Simulation becomes particularly useful when the statistical design or planned analysis is too complex for a suitable closed-form approximation.

Examples include some multilevel models, adaptive procedures, complex missing-data mechanisms, and analyses with nonstandard estimators.

A simulation-based sensitivity analysis requires researchers to specify a data-generating process, repeatedly generate hypothetical datasets, apply the intended analysis, and estimate the frequency with which the predefined success criterion is met.

Simulation results remain conditional on the assumed data-generating models. Greater computational complexity does not automatically make the assumptions more credible.

Can Sensitivity Analysis Address Attrition and Missing Data?

Yes. Researchers can examine how different usable-data rates affect the number of participants who must initially be recruited.

For example, if 200 completed observations are required, a 10% unusable-data rate implies an initial recruitment target of approximately 223 participants, whereas a 20% rate implies 250.

These calculations use the simple relationship between required usable observations and anticipated completion proportions.

However, missing data can affect more than sample size. When missingness depends on participant characteristics or unobserved outcomes, simply recruiting additional participants may not address the resulting bias.

For studies where missing-data mechanisms could materially affect inference, sensitivity analysis may also need to examine the assumptions used in the planned missing-data analysis.

Can Sensitivity Analysis Replace a Defensible Primary Sample-Size Calculation?

No. Sensitivity analysis supplements the primary justification rather than replacing the need to define what the study is intended to establish.

A protocol should ordinarily identify a principal planning objective and explain the assumptions used to determine the recruitment target.

Alternative scenarios then show how the design's operating characteristics change if those assumptions differ from reality.

Without a clear primary objective, a large table of alternative calculations may be difficult to interpret and may not support a coherent recruitment decision.

How Should Sensitivity Analysis Influence the Final Sample Size?

The final decision should consider both the consequences of uncertainty and the scientific importance of the effects or precision targets being examined.

If a modest but meaningful effect is central to the research question, a design with low power at that effect may be difficult to justify, even if it has excellent power for a larger expected effect.

Conversely, designing for an extremely small effect with little substantive importance may impose unnecessary recruitment costs.

Researchers should also consider whether alternative designs could improve efficiency, including repeated measurements, appropriate covariate adjustment, or more reliable outcome instruments.

Sensitivity analysis can reveal a problem with the proposed design, but it does not dictate a single universal solution.

How Should Sensitivity Analysis Be Reported?

A clear report should identify the primary calculation, the alternative parameters examined, the justification for their ranges, and the resulting sample-size or power estimates.

It should also describe how the analysis affected the final sample-size decision.

For a relatively simple study, a concise table may be sufficient. More complex investigations may benefit from power curves, simulation summaries, or supplementary technical documentation.

The calculation method, software, relevant settings, and assumptions should be recorded sufficiently clearly to permit independent reproduction.

Transparency matters more than producing an elaborate sensitivity analysis that does not influence the design.

Testing Whether a Proposed Sample Remains Adequately Powered Under Smaller Effects

These values use normal approximations for an equal-allocation independent-means comparison. They are illustrative and may differ from exact calculations using the noncentral t distribution.

The example illustrates how sensitivity analysis changes the interpretation of a proposed sample. The initial calculation remains valid under its assumptions, but the additional scenarios clarify what the study can and cannot reasonably be expected to detect.

Before recruitment, the researcher must also account for anticipated unusable data and verify that the intended statistical analysis matches the assumptions of the calculations.

Misconceptions About Sample-Size Sensitivity Analysis

Use Sensitivity Analysis When It Can Change a Study-Design Decision

Not every study needs an extensive simulation or a large table of alternative calculations. The appropriate level of analysis depends on how uncertain the planning assumptions are and whether reasonable alternatives would change the recruitment decision.

For a simple design with strong prior evidence and little consequential uncertainty, a concise sensitivity check may be sufficient. For a complex or resource-intensive study, more detailed evaluation may be warranted.

For estimation-focused studies, sensitivity analysis should examine expected confidence-interval precision rather than automatically applying a superiority-test power framework. The choice between precision-based and power-based sample-size planning should follow the primary research objective.

When planning a prevalence survey, uncertainty about expected prevalence can similarly be evaluated across plausible proportions and precision targets.

What matters is whether the analysis provides information that improves the design decision. An elaborate sensitivity table that is ignored when selecting the final sample size offers little practical value.

Before Finalizing Your Sample-Size Sensitivity Analysis

Frequently Asked Questions About Sample-Size Sensitivity Analysis

A Defensible Sample Size Should Survive Reasonable Scrutiny of Its Assumptions

How to Cite This Guide

This guide is intended to be read, shared, and used in research, teaching, and academic work. If you draw on its ideas, explanations, or other content, please acknowledge the source by citing the guide. Doing so gives appropriate credit and helps your readers locate the original resource.