Should Exact Tests and Small-Sample Statistical Methods Influence Research Design Planning?

Should You Choose Your Statistical Method Before Recruiting a Small Sample?

Suppose you are planning a quantitative study involving 24 eligible participants. You want to compare the proportion achieving a particular outcome between two groups, but some expected cell counts may be very small.

You initially plan to use a Pearson chi-square test. A colleague suggests Fisher's exact test instead, arguing that it is more suitable for small samples. Another recommends a permutation test or a penalized regression model.

Should you decide among these methods only after examining the data? Or should the likely statistical limitations influence how you allocate participants, measure outcomes, and calculate the required sample size?

Small-sample analysis is not merely a technical decision made after data collection. Some methods require particular sampling or assignment structures, and the available information may determine whether the intended research question can be answered at all.

Plan Small-Sample Methods Before Data Collection Whenever Feasible

How Small-Sample Statistical Methods Affect Research Design Decisions

What are exact statistical tests?

Exact statistical tests calculate probabilities using a specified sampling distribution rather than relying entirely on an asymptotic approximation.

For example, Fisher's exact test uses a conditional distribution for a contingency table under its null hypothesis and sampling assumptions.

Exact binomial tests use the binomial distribution to evaluate hypotheses about a probability under independent Bernoulli sampling.

The word exact refers to a mathematical property of the test under its assumptions. It does not mean that the study's measurements, sampling process, or causal conclusions are automatically correct.

Exact procedures may be particularly useful when sparse observations make familiar normal or chi-square approximations unreliable.

Does every small study require an exact test?

No. The suitability of a statistical procedure depends on the design and outcome, not simply the number of participants.

Some conventional methods can perform reasonably well with small samples when their assumptions are credible. Other methods may perform poorly even with moderately large samples if events are exceptionally rare or observations are highly dependent.

For example, a paired comparison of approximately normally distributed differences may be appropriately analyzed using a paired t-test with a small number of independent pairs.

Conversely, a large dataset containing only a handful of outcome events may present serious difficulties for conventional logistic regression.

Researchers should evaluate the information structure and inferential target rather than applying an arbitrary sample-size threshold.

When is Fisher's exact test appropriate?

Fisher's exact test is commonly used to examine association in a two-by-two contingency table, particularly when expected cell counts are small.

Its conventional formulation conditions on the observed margins and evaluates tables compatible with those margins under the null hypothesis.

However, the test's interpretation depends on the underlying sampling design and the precise definition of the two-sided p-value.

It is not automatically the best choice for every small categorical dataset.

For example, paired binary outcomes require a method that recognizes pairing rather than an ordinary independent-groups Fisher test.

Similarly, clustered categorical observations cannot generally be treated as independent individuals.

The planned sampling and dependence structure must therefore be considered before selecting the test.

What is the difference between Fisher's exact test and a chi-square test?

A Pearson chi-square test generally uses an asymptotic reference distribution. Fisher's exact test uses a conditional exact distribution under its assumptions.

The choice should not be reduced to the familiar rule that Fisher's test is mandatory whenever an expected count falls below five. Such rules are approximations to a more complicated assessment of test performance.

Can exact tests have low statistical power?

Yes. Exact tests may have limited power when the sample is small and the set of possible outcomes is discrete.

In some settings, the attainable p-values are relatively coarse. A small change in the observed data can produce a substantial change in the p-value.

Conventional conditional exact tests may also be conservative, meaning their actual type I error probability can fall below the nominal level.

Researchers should therefore evaluate the power of the particular exact procedure during sample size planning.

A test that controls false-positive decisions appropriately may still have little probability of detecting a scientifically meaningful effect.

Statistical validity and statistical informativeness are different properties.

What are exact confidence intervals?

Exact confidence intervals are constructed using procedures whose coverage properties are derived from specified finite-sample distributions or test inversions.

For example, the Clopper-Pearson interval is a familiar exact binomial confidence interval.

Such intervals can provide at least nominal coverage under the binomial assumptions, although they may be conservative and wider than some alternatives.

Researchers should distinguish exact coverage guarantees from narrowness or usefulness.

A very wide interval may correctly communicate that the available observations cannot distinguish among substantially different parameter values.

Choosing an exact interval does not remove the information limitation.

How can zero events affect a small study?

When no events are observed, researchers may be tempted to conclude that the event cannot occur.

That conclusion is not justified when the sample is finite and the event probability is uncertain.

For example, if no events occur among 20 independently observed participants, the observed event proportion is zero, but the population event probability may still be positive.

An exact one-sided 95% upper confidence bound under a binomial model can be calculated as follows:

The example shows that observing no events among 20 participants does not establish that the underlying risk is zero.

It also illustrates why precision should be considered before collecting data.

What are permutation and randomization tests?

Permutation tests evaluate a statistic against a reference distribution obtained by rearranging labels or observations according to an assumed exchangeability structure.

Randomization tests use the actual treatment-assignment mechanism to evaluate outcomes under a specified null hypothesis.

For example, a randomized experiment assigning ten participants to each of two conditions may support a randomization-based test consistent with the assignment procedure.

However, arbitrary permutation of labels is not valid when observations are paired, clustered, or otherwise nonexchangeable.

Researchers must preserve the relevant restrictions of the design.

A permutation test is not automatically exact simply because software generates many permutations. Monte Carlo approximation introduces additional computational uncertainty unless the full reference distribution is enumerated.

Why should randomization procedures be planned with the analysis?

Randomization determines which treatment assignments were possible and their probabilities.

These features can affect the appropriate randomization-based analysis.

For example, a trial using blocked randomization should not automatically be analyzed as though every unrestricted allocation was equally possible.

Similarly, a cluster-randomized trial has independent assignment units at the cluster level, even if many individuals contribute outcomes.

Researchers should document the assignment mechanism and ensure that the planned inferential procedure respects it.

This connection between design and analysis is particularly important when few units are randomized.

What happens when logistic regression encounters separation?

Separation can occur when combinations of predictors perfectly or nearly perfectly distinguish outcome categories.

In such settings, conventional maximum likelihood logistic regression may produce infinite or unstable coefficient estimates.

Small samples and rare events can increase the likelihood of this problem.

Penalized likelihood methods, including Firth-type bias reduction, may produce finite estimates in some separation settings.

However, these methods do not establish that the data support a precise or causally valid association.

Researchers should plan model complexity and essential covariates before data collection rather than assuming a penalized estimator can accommodate unlimited predictors.

Are penalized regression and exact regression interchangeable?

No. Exact logistic regression and penalized likelihood approaches use different inferential principles.

Exact methods can be useful for certain sparse categorical data structures but may become computationally demanding or difficult to interpret in more complex models.

Penalized methods modify the estimation objective to reduce instability or bias under specified assumptions.

Neither approach is universally preferable.

The choice depends on the outcome, estimand, predictor structure, available information, and inferential requirements.

Researchers should also consider whether a simpler analysis would better address the research question.

Can small-sample corrections improve clustered analyses?

Some statistical procedures rely on having a sufficiently large number of independent clusters rather than merely a large number of individual observations.

For example, a study involving six schools and hundreds of students may still have limited information for estimating between-school intervention effects.

Conventional cluster-robust standard errors can perform poorly when the number of clusters is small.

Small-sample corrections, appropriately designed randomization inference, or other methods may improve inferential calibration in selected circumstances.

However, these approaches cannot create additional independent schools.

Researchers should base planning on the actual unit of assignment and dependence rather than the total number of records.

Should small-sample methods influence outcome selection?

They may inform whether an outcome can be estimated usefully, but researchers should not redefine outcomes solely to obtain a convenient statistical test.

For example, converting a continuous outcome into a binary variable can discard information and may worsen small-sample limitations.

Conversely, a binary outcome may be scientifically necessary when the research question concerns a meaningful event threshold.

Researchers should choose outcomes according to construct validity and substantive importance.

The planned analysis can then inform measurement precision, sample size, and feasibility.

Statistical convenience should not override the scientific definition of the outcome.

How should sample size be calculated for an exact test?

Sample size planning should reflect the actual test, anticipated effect size, allocation ratio, event probabilities, and desired operating characteristics.

For discrete exact tests, power may change in steps rather than increasing smoothly with sample size.

Consequently, a conventional normal-approximation calculation may not accurately represent the power of the planned exact procedure.

Exact enumeration or simulation may be appropriate for evaluating candidate sample sizes.

Researchers should also examine precision, particularly when estimation rather than hypothesis testing is the primary objective.

The calculation should be completed before recruitment where feasible.

Can a small study be valid but inconclusive?

Yes. A study can use an appropriate design and valid statistical procedure while providing insufficient information for a decisive conclusion.

For example, an exact test may yield a large p-value because the data are compatible with a wide range of effects.

That result does not establish that the treatment groups are equivalent.

Likewise, a wide confidence interval may include meaningful benefit and meaningful harm.

Researchers should distinguish an inconclusive result from a methodological failure.

The study's value depends partly on whether the remaining uncertainty was anticipated and whether the evidence is useful for the intended scientific purpose.

When should researchers change the design rather than the test?

Changing the statistical method may be insufficient when the study lacks a credible comparison group, has too few independent units, or cannot measure essential variables reliably.

For example, an exact test cannot eliminate confounding in a nonrandomized comparison.

Nor can a penalized regression model compensate for substantial selection bias or an outcome definition that does not measure the intended construct.

Researchers may need to reconsider recruitment, allocation, measurement, repeated observations, or the estimand.

Where the entire population is small, review research design options for small populations before assuming that a specialized test is the only necessary modification.

Good design remains the foundation of credible statistical inference.

Planning a Small Randomized Study with a Binary Outcome

Suppose the observed results are eight successful completions among twelve students assigned to interface A and five among twelve assigned to interface B.

The observed difference is substantial in percentage-point terms, but the sample is small and the uncertainty may be considerable.

Fisher's exact test may be appropriate for the planned independent binary comparison, but it does not by itself establish whether the difference is practically important or estimated precisely enough to guide decisions.

The researchers should interpret the effect estimate and uncertainty alongside the design rather than relying exclusively on the p-value.

Common Misconceptions About Exact Tests and Small-Sample Methods

Choose the Design and Analysis Together

Before recruitment, identify the independent units, outcome scale, expected event frequencies, and scientific effect of interest.

Then evaluate whether the planned statistical method provides appropriate error control and useful precision under realistic sample sizes.

For related decisions, examine quantitative research with small populations and whether small studies should prioritize estimation over significance testing.

The analysis plan should help determine what data the study needs, not merely provide a procedure for processing whatever data happen to be collected.

Before Planning a Small-Sample Statistical Analysis

Frequently Asked Questions About Exact Tests and Small Samples

Small-Sample Statistical Methods Should Inform Design, Not Rescue It Afterwards

How to Cite This Guide

This guide is intended to be read, shared, and used in research, teaching, and academic work. If you draw on its ideas, explanations, or other content, please acknowledge the source by citing the guide. Doing so gives appropriate credit and helps your readers locate the original resource.