Manuel B. Garcia

Manuel B. Garcia serves as the Senior Director for Educational Technology and Digital Learning at FEU Institute of Technology, Manila, Philippines. Read More

Contact Info

1607, FEU Tech Building,
P. Paredes St, Sampaloc,
Manila, Philippines
mbgarcia@feutech.edu.ph

Follow Me

Part 04 · Guide 691 of 1272

When Is Precision a Better Sample-Size Planning Target Than Statistical Significance?

Statistical significance may answer whether an effect differs from a null value without establishing its magnitude accurately. Learn when estimation precision is the more informative sample-size target, how to recognize research questions that require it, and how to plan accordingly.

By Manuel B. Garcia
691
When Precision Matters More Than Statistical Significance Guide 691 of 1272
01 · The Question

What If Detecting a Statistically Significant Effect Is Not Enough?

Suppose a study finds that an instructional intervention significantly improves student achievement. The estimated improvement is five points, but its 95% confidence interval ranges from one to nine points.

The result may provide evidence against a zero-effect null hypothesis. Yet it leaves considerable uncertainty about the magnitude of the improvement. An effect of one point might be too small to justify implementation, while an improvement of nine points could be highly consequential.

If the purpose of the research is to inform an institutional decision, obtaining a statistically significant result may not be sufficient. Researchers may need an estimate precise enough to distinguish effects with different practical implications.

When should sample-size planning prioritize that precision rather than the probability of achieving statistical significance?

02 · The Short Answer

Prioritize Precision When the Magnitude of the Effect Matters More Than Rejecting a Null Hypothesis

In Brief

Precision is generally a better primary sample-size planning target than statistical significance when the research objective is to estimate the magnitude of a population parameter or effect accurately enough to support scientific interpretation or practical decisions. In such studies, sample size should be justified using a meaningful confidence-interval width or another appropriate estimation-accuracy criterion.

Statistical power remains appropriate for studies designed primarily to test prespecified hypotheses. Precision and power are related but answer different questions, and some research designs need to satisfy both criteria.

03 · What You Need to Know

Recognizing When Estimation Precision Should Drive Sample-Size Planning

Why Is Statistical Significance Sometimes an Inadequate Research Target?

Statistical significance concerns the compatibility of observed data with a specified null hypothesis under a statistical model. For a conventional two-sided test, a p-value below a prespecified significance level may lead researchers to reject the null hypothesis.

However, the p-value does not directly describe the size of an effect, its practical importance, or the precision with which it has been estimated.

With sufficiently large samples, effects too small to matter practically may be statistically distinguishable from zero. Conversely, a study with a small sample may fail to reject a null hypothesis even when its data remain compatible with substantively important effects.

The American Statistical Association's statement on p-values emphasizes that statistical significance does not measure effect size or the importance of a result.

Consequently, when the research question concerns how large an effect is, rather than merely whether a null value can be rejected, estimation precision may provide a more relevant planning criterion.

What Does It Mean to Prioritize Precision?

Precision-based planning begins by specifying how much uncertainty around an estimate is acceptable.

For example, a researcher might want to estimate an average improvement to within approximately two examination points, or a population prevalence to within three percentage points.

These requirements can be translated into confidence-interval width targets and used to calculate the sample size needed under specified assumptions.

The objective is not necessarily to obtain a confidence interval excluding zero. It is to obtain an interval sufficiently informative for the intended interpretation.

Determining an acceptable confidence interval width requires substantive judgment about the outcome scale and the distinctions the research must resolve.

When Is Precision Particularly Important?

Precision is often the more direct planning target when researchers aim to estimate a population characteristic, describe the magnitude of an association, or quantify an effect for a subsequent decision.

Consider a study estimating the average weekly workload of university teachers. A hypothesis test against an arbitrary workload value may contribute little if the real objective is to determine the average workload with sufficient accuracy for institutional planning.

Similarly, researchers studying the prevalence of academic burnout may be primarily interested in obtaining a useful population estimate rather than testing whether prevalence differs from an arbitrary percentage.

Precision is also valuable when an intervention's cost or consequences depend on the magnitude of its effect. In such cases, knowing that the effect is statistically distinguishable from zero may not resolve whether implementation is worthwhile.

How Do Precision and Statistical Significance Differ?

Consideration Statistical significance Estimation precision
Central question Does the evidence support rejecting a specified null hypothesis? How narrowly can the population parameter be estimated?
Common reporting measure Test statistic and p-value Effect estimate and confidence interval
Sample-size criterion Desired power at a specified alternative Desired interval width or estimation accuracy
Primary strength Supports prespecified hypothesis testing. Quantifies uncertainty about parameter magnitude.
Important limitation Significance does not establish practical importance. Precision does not establish absence of systematic bias.

These approaches are not mutually exclusive. Confidence intervals can support hypothesis testing, and adequately powered studies can provide useful estimates.

The difference lies primarily in which objective determines the required sample size.

Why Can a Significant Result Still Be Too Imprecise?

A statistically significant result can have a confidence interval that excludes zero but spans a broad range of effect magnitudes.

For example, an estimated treatment benefit of five points with a 95% confidence interval from 0.5 to 9.5 points is statistically distinguishable from zero under a corresponding conventional two-sided test.

Nevertheless, the interval includes effects that may be practically negligible and effects that may justify substantial investment.

If the intended decision depends on distinguishing those possibilities, the study may not have achieved sufficient estimation precision.

Statistical significance and practical usefulness should therefore be evaluated separately.

How Does the Smallest Effect Size of Interest Help?

A smallest effect size of interest (SESOI) identifies the minimum effect considered substantively important for a particular research objective.

Suppose an improvement of four examination points is the smallest benefit that would justify implementing a new learning intervention.

An estimated effect of five points with a confidence interval from one to nine points does not establish that the population improvement exceeds four points.

Even though the point estimate exceeds the threshold, the interval remains compatible with smaller improvements.

A SESOI can therefore help researchers determine what precision is needed. However, it does not automatically specify the required interval width.

If the objective is to demonstrate that an effect exceeds the SESOI, a threshold-focused hypothesis test or assurance-based interval criterion may be more appropriate than simply targeting a narrow expected interval.

How Is Sample Size Calculated for Precision?

For estimating a population mean from independent observations, with an assumed population standard deviation and a suitable normal approximation, the required sample size is:

Approximate Precision-Based Sample Size
n ≈ (Z1−α/2 × σ / E)2

n = required independent observations; Z1−α/2 = normal critical value for the chosen two-sided confidence level; σ = assumed population standard deviation; E = desired confidence-interval half-width in the outcome's original units; α = one minus the confidence level.

Suppose the researcher wants to estimate a mean with a 95% confidence interval having an approximate half-width of two points. The assumed population standard deviation is ten points.

n ≈ (1.96 × 10 / 2)2 = 96.04.

Rounding upward gives 97 independent observations under the normal approximation.

The result targets approximate precision, not a probability of obtaining a statistically significant result. A t-based interval using an estimated standard deviation may require a different sample size.

Different estimands require different formulas. For a proportion, anticipated prevalence influences the variance. For a correlation, Fisher's transformation may support precision planning under suitable assumptions.

Researchers should not apply the single-mean formula to a between-group treatment effect without accounting for the variance of the group difference.

How Does Precision-Based Planning Compare With Power-Based Planning?

Consider two studies using the same continuous outcome.

One aims to estimate a population mean within two points. The other aims to detect a five-point difference between independent groups with 80% power at a two-sided significance level of 0.05.

The first requires a precision-based calculation for one mean. The second requires a power calculation for the difference between two means.

Even if the calculations produce similar sample sizes, they do not answer the same question.

The broader distinction between precision-based and power-based sample-size planning concerns the intended inference rather than a preference for one statistical technique.

Can Precision Be More Important in Exploratory Research?

Yes, particularly when an exploratory study seeks to characterize a poorly understood parameter rather than test a specific theoretical prediction.

For example, researchers investigating a newly developed educational assessment may initially want to estimate score variability, associations with established measures, or other measurement properties.

Confidence intervals can communicate the uncertainty surrounding these preliminary estimates more directly than a collection of significance tests.

However, exploratory status does not automatically justify a small sample. A study intended to provide useful preliminary estimates still needs an appropriate rationale for its precision.

When resources constrain the sample, researchers should report the precision that the feasible design can realistically achieve.

When Might Precision Be More Important in Evaluation Research?

Evaluation studies often inform decisions about whether an intervention, program, or policy should continue, expand, or change.

The magnitude of an effect may be more relevant to those decisions than whether the effect differs statistically from zero.

Suppose a digital learning intervention costs substantially more than existing instruction. A statistically significant improvement of 0.5 examination points may not justify adoption, whereas an improvement of five points might.

Researchers may therefore need an interval sufficiently narrow to distinguish effects with different practical consequences.

Nevertheless, evaluation research may also require formal hypothesis testing, including superiority, noninferiority, or equivalence analyses. Precision does not automatically replace these inferential objectives.

Does Precision-Based Planning Mean Statistical Significance Should Be Ignored?

No. Hypothesis tests can remain relevant when the research question requires them.

The concern is with allowing statistical significance to become the sole criterion of scientific value.

Researchers can report estimates, confidence intervals, and appropriately specified hypothesis tests together. The primary sample-size justification should correspond to the study's central objective.

For example, a precision-focused study may report whether a confidence interval excludes zero while emphasizing the estimated magnitude and its uncertainty.

Conversely, a confirmatory trial may be powered for a prespecified hypothesis while still requiring careful interpretation of the confidence interval.

Does a Narrow Confidence Interval Guarantee an Accurate Result?

No. Statistical precision and accuracy are not identical.

A large but systematically unrepresentative sample can produce a narrow interval around a biased estimate.

Measurement error, confounding, model misspecification, and selection mechanisms may also affect the validity of an estimate.

Precision-based planning addresses one important component of uncertainty. It does not substitute for appropriate sampling, valid measurement, or a defensible analytical strategy.

Watch Out

Do not interpret a narrow confidence interval as proof that an estimate is correct. A precise estimate of the wrong population, an inadequately measured construct, or a biased association may still lead to misleading conclusions.

What If the Realized Confidence Interval Is Wider Than Planned?

Conventional precision-based calculations usually target an approximate or expected interval width under specified assumptions.

The realized width may differ because the observed variance, estimated proportion, or other relevant quantities are random.

If a study requires a specified probability of obtaining an interval no wider than a particular threshold, researchers may need an assurance-based precision calculation.

Such planning distinguishes the expected width from the probability of achieving the desired width.

This distinction becomes particularly relevant when the width requirement is strict or when the planning variance is uncertain.

How Should Researchers Handle Uncertain Variability?

Precision calculations depend on assumptions about the variability of the estimator.

When the population standard deviation or other variance component is uncertain, researchers should use relevant previous evidence and evaluate plausible alternatives.

A sample-size sensitivity analysis can show how the required sample changes under different variability or precision assumptions.

This is preferable to presenting one apparently exact requirement based on an unsupported estimate.

Can Precision and Power Be Joint Planning Requirements?

Yes. Some studies need to test a prespecified hypothesis and estimate the effect with sufficient precision for interpretation.

For example, a confirmatory intervention study may require 80% or 90% power to test superiority while also aiming for an interval narrow enough to inform implementation decisions.

Researchers can calculate the requirements under both criteria and select a design that adequately addresses the prespecified objectives.

However, it is important to distinguish a fixed-width objective from a probability of excluding a particular threshold. These require different planning calculations.

Maxwell, Kelley, and Rausch (2008) discuss the distinction between statistical power and accuracy in parameter estimation and demonstrate why both can matter for sample-size planning.

04 · A Practical Example

Why an Educational Evaluation May Need Precision Rather Than Significance Alone

Hypothetical Example

Evaluating the Improvement Produced by a Learning Platform

A university evaluates a digital learning platform using a randomized comparison with conventional instruction. The outcome is a continuous achievement score.

The research team considers an average improvement of four points the smallest benefit that would justify substantial implementation costs. The assumed common within-group standard deviation is ten points.

For illustration, the team compares a conventional superiority power calculation with a precision-based calculation for estimating the between-group mean difference.

Approximate Precision for an Independent-Groups Mean Difference
E ≈ Z1−α/2 × σ × √(2/n)

E = approximate confidence-interval half-width for the mean difference; Z1−α/2 = normal critical value for the confidence level; σ = assumed common within-group standard deviation; n = number of independent participants per group under equal allocation.

At 95% confidence, σ = 10, and n = 63 participants per group:

E ≈ 1.96 × 10 × √(2/63) ≈ 3.49 points.

The planned total confidence-interval width is therefore approximately 6.98 points.

If the team instead requires an approximate half-width of two points, rearranging gives n ≈ 2 × (1.96 × 10 / 2)2 = 192.08, rounded upward to 193 participants per group.

These are normal-approximation results, not exact t-based or assured-width calculations.

Step 1: Identify the hypothesis-testing requirement

For a target difference of five points, 80% power, and two-sided α = 0.05, the conventional normal approximation gives approximately 63 participants per group.

Step 2: Examine the resulting precision

With 63 participants per group, the anticipated 95% confidence-interval half-width is approximately 3.49 points.

Step 3: Compare the interval with the decision threshold

If the observed improvement is five points, an illustrative interval using the planned half-width would span approximately 1.51 to 8.49 points. It would include improvements below and above the four-point meaningfulness threshold.

Step 4: Calculate a narrower precision target

A target half-width of two points requires approximately 193 participants per group under the stated normal approximation. An illustrative interval centered at five points would then span three to seven points, still crossing the four-point threshold.

Step 5: Choose the appropriate inferential objective

If the goal is to estimate the treatment effect with an expected half-width of two points, precision-based planning is appropriate. If the goal is to establish that the benefit exceeds four points with a specified probability, the team should instead calculate sample size for that threshold-focused objective.

The example demonstrates that increasing precision can improve the usefulness of an estimate without necessarily guaranteeing a particular decision.

A confidence interval's position depends on the observed effect, not only its width. Researchers should therefore distinguish a target for estimation accuracy from a requirement to establish that the effect exceeds a prespecified threshold.

05 · What Researchers Often Get Wrong

Misconceptions About Precision and Statistical Significance

Misconception

A Statistically Significant Result Is Automatically Informative

Statistical significance does not establish practical importance or precise estimation. A significant effect may be too small to matter or estimated with substantial uncertainty about its magnitude.

Misconception

A Nonsignificant Result Means the Effect Is Absent

Failure to reject a null hypothesis does not establish that the population effect is zero. A wide confidence interval may remain compatible with substantively important effects.

Misconception

Precision-Based Planning Is Only for Descriptive Surveys

Precision can be a primary objective when estimating intervention effects, correlations, regression coefficients, or other parameters whose magnitudes matter for scientific interpretation.

Misconception

A Narrow Interval Proves That an Effect Is Meaningful

A narrow interval can be centered on a negligible effect. Meaningfulness depends on the estimated magnitude, uncertainty, and substantive decision criteria.

Misconception

Precision-Based Planning Eliminates the Need for Statistical Assumptions

Precision calculations depend on assumptions about variability, sampling, the estimator, and the confidence-interval procedure. These assumptions require justification and sometimes sensitivity analysis.

Misconception

A Planned Interval Width Guarantees a Decision

Expected interval width does not determine where the realized interval will lie. Establishing that an effect exceeds a threshold requires an appropriately specified inferential criterion.

06 · What This Means for You

Choose Precision When the Research Must Quantify Rather Than Merely Detect

Before selecting a sample-size formula, identify the conclusion that the study needs to support.

If the primary objective is to describe the magnitude of a parameter with useful accuracy, define a precision target. If the objective is to test a specific hypothesis, determine the appropriate power requirement. If both matter, evaluate both.

A simple decision framework

If the question asks how large a population parameter or association is
Prioritize an appropriate estimation-precision criterion.
If the question asks whether a prespecified hypothesis should be rejected
Use a power calculation corresponding to the planned statistical test.
If practical decisions depend on effect magnitude
Specify meaningful effect thresholds and evaluate whether the planned interval precision will be informative.
If the objective is to demonstrate that an effect exceeds a threshold
Use a threshold-focused test or appropriate probability-based interval criterion rather than expected width alone.
If recruitment capacity is fixed
Calculate the achievable precision and assess whether the resulting uncertainty is acceptable.

For estimation of a continuous outcome, the sample size for estimating a population mean provides one common application.

For association research, precision-based planning for correlation estimation may be more informative than designing solely to reject a zero-correlation hypothesis.

The final protocol should state the primary estimand, the required precision or power, the planning assumptions, and why the selected criterion addresses the research question.

07 · A Quick Checklist

Before Prioritizing Precision Over Statistical Significance

Before finalizing your sample-size justification, check:
Identify whether the primary objective is estimation, hypothesis testing, or both.
Define the population parameter or effect that must be estimated.
Specify what differences in parameter magnitude would change scientific interpretation or practical decisions.
If using precision-based planning, justify the confidence level and acceptable interval width.
If using power-based planning, justify the target effect, significance level, and desired power.
Distinguish expected interval width from the probability of excluding a substantive threshold.
Verify that the variance assumptions and sample-size formula match the planned estimator and design.
Examine how uncertainty in important planning parameters affects the required sample.
Document why the selected sample will provide information sufficient for the intended research objective.
08 · Frequently Asked Questions

Frequently Asked Questions About Precision and Statistical Significance

Is precision more important than statistical significance?

It depends on the research objective. Precision is often the better primary criterion when estimating magnitude is central. Statistical power is appropriate when testing a prespecified hypothesis is the primary objective.

Can a statistically significant result have a wide confidence interval?

Yes. An interval may exclude the null value while still including effects with substantially different scientific or practical implications.

Can a nonsignificant result still be informative?

Yes. A sufficiently precise interval may rule out effects of meaningful magnitude even when it includes zero. A wide interval, however, may leave important effects unresolved.

Does precision-based sample-size planning require a p-value?

No. Precision-based planning commonly uses a confidence level, target interval width, and relevant variance assumptions. A hypothesis test is not necessarily part of the primary calculation.

Can I use precision-based planning for an experimental study?

Yes. Experiments may be designed to estimate treatment effects with useful precision. The calculation must reflect the variance of the treatment-effect estimator and the actual experimental design.

Does a narrow confidence interval establish practical importance?

No. Practical importance depends on the estimated effect and its substantive consequences. A narrow interval may provide precise evidence that an effect is too small to matter.

Can I plan for both 80% power and a target confidence interval width?

Yes. When both objectives are important, calculate the requirements separately and select a design that appropriately satisfies the prespecified criteria.

Is a precision-based study automatically more rigorous?

No. Rigor depends on alignment among the research question, design, sampling, measurement, analysis, and interpretation. Precision-based planning is advantageous when it matches the intended inference, not simply because it uses confidence intervals.

09 · The Bottom Line

Prioritize the Information the Research Needs, Not Merely a Significant Result

The Bottom Line

Precision is a better primary sample-size planning target than statistical significance when the study needs to estimate the magnitude of an effect or population parameter with enough accuracy to support meaningful interpretation or decisions.

Statistical significance remains useful for appropriate hypothesis tests, but it does not establish effect magnitude or practical importance. Select the planning criterion that matches the research question, and evaluate both precision and power when the intended inference requires them.

10 · Sources and Further Reading

Research Literature on Precision, Statistical Significance, and Sample-Size Planning

11 · Cite this Guide

How to Cite This Guide

This guide is intended to be read, shared, and used in research, teaching, and academic work. If you draw on its ideas, explanations, or other content, please acknowledge the source by citing the guide. Doing so gives appropriate credit and helps your readers locate the original resource.

Has the Field Guide helped your research?

If a guide helped clarify a question, inform a research decision, or move your work forward, I would love to hear about your experience. Your story may also help other researchers discover the Field Guide.

Share Your Experience
Takes only a few minutes