Will the Study Narrow Uncertainty Enough to Matter?

How much uncertainty will actually disappear because of your study?

Research begins with uncertainty, but simply collecting more data does not guarantee that the important uncertainty will shrink.

A study might produce a new estimate while leaving several substantively different possibilities compatible with the evidence. It might measure something very precisely that was never the main source of uncertainty. Or it might improve knowledge slightly without improving it enough to affect any scientific interpretation, practical decision, or subsequent research choice.

So before asking whether a study can produce a result, ask a more demanding question: will the study reduce the uncertainty that matters by enough to make a difference?

A useful study should make an important uncertainty meaningfully smaller

Not every reduction in uncertainty has the same research value

First identify what you are uncertain about

“More research is needed” is not a sufficiently precise description of an evidence gap. Before designing another study, identify what remains uncertain.

You might be uncertain about whether an effect exists, how large it is, which mechanism produces it, whether it generalizes to another population, which intervention performs better, whether an apparent relationship is causal, or which parameter is responsible for uncertainty in a larger model.

These are different uncertainties. A study capable of reducing one may do little about another.

Suppose researchers already have reasonably consistent evidence that an educational intervention produces a positive average effect, but estimates of its magnitude vary substantially. Another small study asking whether the intervention has any effect may add data without resolving the more consequential question of how large the effect is.

Good planning therefore starts by naming the unresolved quantity, explanation, prediction, or decision explicitly.

More information is not necessarily the information you need

Imagine that a decision depends primarily on two uncertain quantities: the effectiveness of an intervention and its implementation cost. Existing evidence already estimates effectiveness quite precisely, while implementation cost varies widely across institutions.

A large new effectiveness study could produce an even more precise estimate. Yet the practical decision might remain uncertain because cost, not effectiveness, is what currently separates the plausible choices.

This distinction is formalized in some decision-analytic settings through value of information analysis. The framework asks how additional information is expected to improve a decision by reducing relevant uncertainty. It can also help identify which uncertain parameters are worth learning more about and whether the expected benefit of additional research justifies its cost.

You do not need to perform a formal value-of-information analysis for every research project. The underlying question is nevertheless useful: which uncertainty is actually preventing a clearer conclusion or better decision?

Precision matters when substantively different possibilities remain plausible

Consider an estimated intervention effect of 4 percentage points.

If the uncertainty surrounding that estimate spans from a meaningful negative effect to a substantial positive effect, the study leaves very different interpretations open. If a more precise estimate places the effect within a much narrower range, some of those possibilities may become difficult to sustain.

This is one reason effect estimates should be interpreted with their uncertainty rather than as isolated point estimates.

Greater precision often improves informativeness, but the concepts are not identical. You can estimate an irrelevant quantity with extraordinary precision.

Ask whether the remaining uncertainty crosses an important boundary

Suppose a school will adopt an intervention only if its benefit is sufficiently large to justify additional costs. Before the study, plausible effect sizes span both sides of that practical threshold.

A useful study could narrow the estimate enough that the evidence lies predominantly on one side of the threshold. A less informative study might shrink the uncertainty somewhat while still leaving substantial support for effects on both sides.

The same logic applies outside decision-making. A theoretical prediction may imply that an effect should be positive rather than negative, increase rather than remain constant, or differ substantially across specified conditions. Research becomes informative when its expected precision is sufficient to discriminate among those consequential possibilities.

This connects uncertainty reduction to defining what an informative result would look like before conducting the study. You need to know which distinctions matter before deciding whether the planned evidence will make them sufficiently clear.

The amount of uncertainty reduction needed depends on what is at stake

There is no universal percentage by which uncertainty must decrease before a study becomes worthwhile.

A modest reduction may be valuable when the information is inexpensive to obtain, feeds into a larger cumulative evidence base, resolves an important parameter, or determines whether a more costly study should proceed. A much larger reduction might be necessary when the study is intended to support a high-stakes decision.

Research value therefore depends partly on the consequences of remaining uncertain.

This is particularly explicit in decision-theoretic value-of-information approaches, where the value of additional evidence depends not simply on how much uncertainty exists but on whether resolving it improves the expected consequences of the decision.

Some uncertainty cannot be fixed by increasing the sample size

When researchers hear “uncertainty,” the immediate response is often to increase the sample. That can reduce sampling uncertainty, but it does not automatically address every important source of uncertainty.

A very large sample cannot rescue a poorly measured construct, establish causality from a design that does not support causal inference, determine whether findings generalize to populations never represented in the study, or distinguish explanations that make the same prediction for the observed data.

Before increasing sample size, diagnose the source of uncertainty. The appropriate response might instead involve stronger measurement, a different comparison group, another population, longitudinal observation, experimental manipulation, additional outcomes, or a fundamentally different research design.

Think about uncertainty before and after the proposed study

A useful planning exercise is to compare two states of knowledge.

Before the study: What possibilities are currently credible?

After the study: Given realistic data quality and precision, which of those possibilities could the evidence make substantially less credible?

If the answer is “almost none,” the study may produce another result without resolving the uncertainty that motivated it.

If several consequential possibilities could be separated, the study has a clearer informational role.

When additional evidence reduces uncertainty but not enough

The four-point threshold is hypothetical and would require substantive justification in a real study. More importantly, this example does not imply that every project should be optimized around a binary decision threshold.

The same reasoning applies to theoretical research. If two explanations imply meaningfully different patterns, the useful study is the one expected to distinguish those patterns with adequate precision.

Why collecting more evidence may leave the important uncertainty intact

Design the study around the uncertainty that blocks progress

Before collecting new data, identify the uncertainty that currently prevents a stronger inference, clearer explanation, better decision, or useful next research step. Then ask whether your proposed design is capable of reducing that particular uncertainty.

For some decision problems, formal value-of-information methods can quantify the expected benefit of additional research and compare it with research costs. In many ordinary research projects, however, a qualitative version of the same reasoning is already valuable.

Ask what you do not know, why knowing it better matters, and how much clearer the proposed evidence is realistically capable of making it. If the answer to the last question is disappointing, finding out before data collection is considerably more useful than discovering it in the limitations section.

Before collecting more evidence, identify what will become clearer

Questions about uncertainty reduction and research value

Reduce the uncertainty that actually prevents a useful answer

How to Cite This Guide

This guide is intended to be read, shared, and used in research, teaching, and academic work. If you draw on its ideas, explanations, or other content, please acknowledge the source by citing the guide. Doing so gives appropriate credit and helps your readers locate the original resource.