03 · What You Need to Know
How to Choose Between Estimation and Hypothesis Testing in Small Studies
What is the difference between estimation and significance testing?
Estimation aims to quantify an unknown parameter or effect, such as a mean difference, risk difference, correlation, or treatment effect.
Statistical significance testing evaluates how compatible the observed data are with a specified null hypothesis under an assumed statistical model or testing procedure.
For example, a study might estimate that an intervention improves assessment performance by six points. A confidence interval would describe the precision of that estimate under the relevant assumptions.
A conventional hypothesis test might instead evaluate a null hypothesis of no difference and produce a p-value.
These approaches are related. In many familiar settings, confidence intervals and hypothesis tests are mathematically connected.
Nevertheless, they emphasize different aspects of the evidence. Estimation foregrounds magnitude and uncertainty, whereas conventional significance testing often emphasizes a decision relative to a specified threshold.
Why can statistical significance be difficult to achieve with small samples?
Small samples generally provide less information about unknown parameters than larger samples under otherwise comparable conditions.
Consequently, estimates may have larger standard errors, and hypothesis tests may have limited power to detect modest effects.
For example, an intervention could produce a meaningful improvement, yet the observed data might remain compatible with no improvement because the estimate is imprecise.
A nonsignificant result therefore does not establish that the intervention has no effect.
Likewise, a statistically significant result from a small study may be associated with substantial uncertainty about the true effect magnitude.
The American Statistical Association's statement on p-values cautions against treating a significance threshold as a complete measure of scientific evidence.
What does an effect estimate tell you?
An effect estimate describes the observed magnitude and direction of a relationship or treatment contrast under a specified estimator.
Examples include a mean difference of five assessment points, a risk difference of three percentage points, or a correlation coefficient of 0.40.
However, the numerical estimate must be interpreted according to the design.
A difference between nonrandomized groups may describe an association rather than a causal effect.
A before-and-after difference may reflect maturation, practice, or other changes occurring alongside the intervention.
Researchers should define the estimand before choosing the estimator and avoid treating every numerical difference as a causal treatment effect.
Why are confidence intervals especially useful?
Confidence intervals communicate the range of parameter values compatible with the data and inferential procedure at a specified confidence level.
Under the relevant frequentist assumptions, a 95% confidence interval procedure has 95% coverage in repeated applications.
It does not mean that a fixed parameter has a 95% probability of lying within the particular interval already calculated.
For example, an estimated treatment benefit of five points with a 95% confidence interval from minus two to twelve points indicates that the data are compatible with a range including modest harm, no difference, and substantial benefit.
That interpretation is more informative than stating only that the result was not statistically significant.
However, confidence intervals reflect the assumptions of the statistical procedure. They do not automatically account for confounding, selection bias, or measurement error.
Can a wide confidence interval still be informative?
Yes, although its usefulness depends on the scientific decision.
A wide interval may show that the study cannot distinguish among several substantively different possibilities.
For example, an interval spanning meaningful benefit and meaningful harm indicates that the evidence is insufficient for a confident decision about the direction of the effect.
That conclusion can be scientifically valuable because it identifies the uncertainty remaining after the study.
However, researchers should not describe an imprecise estimate as strong evidence merely because it has been reported with a confidence interval.
Estimation makes uncertainty visible; it does not eliminate it.
How should practical significance be evaluated?
Practical significance concerns whether an effect is large enough to matter for the substantive research problem.
For example, an improvement of two points on a 100-point assessment may be important in one educational context but negligible in another.
Researchers should define a meaningful effect threshold using relevant theory, prior evidence, stakeholder priorities, measurement properties, and the consequences of decisions.
That threshold should not be selected after examining the observed effect merely to make the findings appear important.
Practical significance and statistical significance are distinct. A precisely estimated trivial effect may be statistically significant, while an important but imprecisely estimated effect may not be.
Should researchers use standardized effect sizes?
Standardized effect sizes may help compare results across measurements with different scales.
For example, a standardized mean difference expresses a group difference relative to a specified standard deviation.
However, standardized effects can be unstable when the standard deviation is estimated from a small sample.
They can also obscure the substantive meaning of the original measurement scale.
Where possible, researchers should report interpretable effects in their original units alongside standardized measures when those measures add value.
Small-sample bias corrections, such as those associated with Hedges-type standardized mean differences, may be relevant in appropriate settings.
Such corrections do not resolve confounding or guarantee precise estimates.
What is precision-based sample size planning?
Precision-based planning selects a sample size according to the desired uncertainty around an estimate rather than only the probability of rejecting a null hypothesis.
For example, researchers may aim to estimate a mean with a confidence interval no wider than a specified number of points.
Under simple independent-sampling assumptions, the approximate margin of error for a mean depends on the standard deviation and sample size.
When the eligible population is fixed and small, the desired precision may be unattainable.
Researchers should report that limitation rather than claiming that the available population satisfies a target it cannot achieve.
Does a census eliminate uncertainty?
A complete census can eliminate sampling uncertainty for a fully observed descriptive quantity in a fixed finite population.
For example, if all 26 eligible students complete an assessment, their mean score is known for that defined cohort, assuming accurate measurement.
However, uncertainty may remain when researchers seek to generalize to future cohorts, estimate causal effects, or account for measurement error.
Researchers should distinguish finite-population description from broader inferential objectives.
The distinction is particularly important when considering recruitment of the entire eligible population.
When does formal hypothesis testing remain appropriate?
Hypothesis testing may remain appropriate when the study has a clearly specified hypothesis and a decision rule relevant to its scientific or practical purpose.
For example, a randomized experiment may test a sharp null hypothesis using an inference procedure consistent with its assignment mechanism.
Equivalence and noninferiority studies also involve formal inferential objectives, although they require carefully justified margins and adequate information.
However, researchers should not conduct a conventional significance test merely because it is expected in a thesis or journal article.
The test should answer a meaningful question and be supported by the design.
Even when hypothesis testing is primary, effect estimates and uncertainty should ordinarily be reported.
Can a nonsignificant result establish equivalence?
No. Failure to reject a null hypothesis of no difference does not establish that two interventions are equivalent.
Equivalence requires a prespecified range of differences considered sufficiently small to be practically unimportant.
Researchers must then use an appropriate inferential procedure to evaluate whether the evidence supports equivalence within that margin.
Small samples often make equivalence difficult to establish because confidence intervals may be too wide.
Similarly, noninferiority requires a justified margin and an appropriate design and analysis.
A nonsignificant superiority test should never be relabeled as evidence of equivalence after the results are known.
What is the problem with interpreting only p-values?
A p-value does not directly measure the probability that the null hypothesis is true.
Nor does it measure the magnitude, practical importance, or probability of replication of an effect.
Small studies may produce highly variable p-values because individual observations can exert considerable influence.
Researchers should avoid dividing results into scientifically meaningful and meaningless categories solely according to whether p falls below 0.05.
The ASA's broader discussion of moving beyond p-value thresholds encourages interpretation based on effect sizes, uncertainty, study design, and substantive context.
This does not require abandoning hypothesis tests. It requires avoiding conclusions that exceed what those tests establish.
Can Bayesian estimation be useful?
Bayesian methods can express uncertainty through posterior distributions and credible intervals.
For example, researchers may estimate the posterior probability that a treatment effect exceeds a prespecified meaningful threshold.
However, posterior conclusions depend on the likelihood, prior distribution, and model assumptions.
When the sample is small, prior information may exert substantial influence.
Researchers should justify priors and examine sensitivity to plausible alternatives.
The role of Bayesian designs for small samples should be evaluated according to the research objective rather than treated as an automatic solution to low power.
How should estimation be planned before data collection?
Researchers should specify the estimand, outcome scale, effect measure, and inferential population.
They should identify the smallest effect of substantive interest or the precision needed for the intended decision.
Where the population size is fixed, they should evaluate the range of precision likely to be achievable under plausible variability and missingness assumptions.
The analysis plan should also address potential outliers, measurement reliability, and relevant confounding.
Planning should occur before outcome results are examined whenever feasible.
Otherwise, researchers risk selecting whichever estimate, scale, or inferential procedure makes the observed findings appear most favorable.
What if the confidence interval is too wide to support a conclusion?
The appropriate response is to report the uncertainty accurately.
Researchers may explain that the available data are compatible with several substantively different effects and therefore cannot resolve the original question decisively.
They should not reinterpret a wide interval as evidence of no effect.
Nor should they select a different statistical test solely to obtain a significant result.
If the study is still in the planning stage, alternative designs, additional measurement occasions, or collaboration may improve information under suitable conditions.
However, those changes should preserve the scientific question and respect the relevant design assumptions.
How should small-study findings be reported?
Reports should identify the eligible population, achieved sample, missing data, effect measure, and uncertainty interval.
Researchers should explain whether the estimate describes the observed finite population or supports inference to a broader population.
Where relevant, they should compare the interval with a prespecified threshold of practical importance.
Claims should distinguish observed associations from causal effects and avoid describing nonsignificant results as evidence of absence.
Transparent estimation may be particularly valuable in quantitative studies involving small populations, where conventional significance decisions may conceal substantial uncertainty.
