What Would an Informative Result Look Like Before You Conduct the Study?

What would your result actually tell you?

You have a research question, a design, variables to measure, and perhaps even a statistical analysis plan. There is still another question worth asking before you collect a single observation: what would an informative result actually look like?

Researchers often imagine the result they hope to obtain. Perhaps an intervention improves learning, two groups differ, an association appears, or a hypothesized mechanism receives support. That is understandable, but it is not quite the same as designing an informative study.

A stronger test is to imagine several plausible outcomes, including results that are smaller than expected, close to zero, opposite in direction, or substantially uncertain. Then ask what each would allow you to learn.

If you cannot explain how realistic results would change your understanding of the research question, the problem may lie upstream in the question, measurement, design, or precision of the planned study.

An informative result changes what you can reasonably conclude

Think about possible conclusions before thinking about desirable results

Start with the uncertainty the study is supposed to resolve

Research usually begins because something important remains uncertain. Perhaps two theories make different predictions. Perhaps previous studies disagree about the size of an effect. Perhaps a university does not know whether a new instructional intervention produces enough improvement to justify adopting it.

An informative study should make at least some of those possibilities easier to distinguish.

This shifts the planning question from “What result do I expect?” to “What would I learn under different plausible results?” The distinction matters because an expected result can be scientifically unsurprising while an unexpected result can be highly informative.

For example, suppose previous evidence suggests that a teaching intervention may improve examination performance. Simply observing a positive difference does not settle very much. A difference of 0.2 percentage points, 3 percentage points, and 15 percentage points are all positive, yet they could have very different theoretical and practical implications.

Do not define informative as statistically significant

Statistical significance answers a particular inferential question under a statistical model. It does not determine whether the estimated effect is large enough to matter, whether the estimate is precise enough for the intended conclusion, or whether the result distinguishes between scientifically relevant possibilities.

A very precisely estimated trivial effect can be statistically significant. Conversely, an estimate that is not conventionally significant may still rule out effects large enough to matter, depending on the design and inferential framework.

This is one reason researchers may define a smallest effect size of interest before examining the data. The idea is not that every study needs one universal numerical threshold. Rather, the researcher asks how large an effect would need to be before it becomes theoretically, practically, clinically, educationally, or otherwise consequential. Equivalence-testing approaches can then evaluate whether effects at least that large can reasonably be ruled out.

Think in ranges rather than one hoped-for number

You rarely know in advance that the result will be exactly 0.31, a five-point difference, or an odds ratio of 1.6. Planning around one anticipated value can therefore give a false sense of precision.

Instead, sketch a set of substantively different result regions. The boundaries depend on the question, but a study might plausibly produce an effect that is meaningfully positive, positive but too small to matter, approximately negligible, meaningfully negative, or too uncertain to classify.

The point is not to predict which region will occur. It is to decide what each region would mean.

Precision determines which conclusions your study can support

Suppose your estimate is a two-point improvement. That number alone does not tell you how informative the study was. An estimate of two points with substantial uncertainty could remain compatible with a meaningful benefit, no meaningful effect, or even harm. A much more precise estimate centered on two points may permit stronger discrimination among those possibilities.

This is why sample-size planning should not be reduced to the probability of obtaining p <.05. Conventional power calculations can be useful, but researchers may also need to consider the precision of effect estimates and the consequences of noisy estimates. Design analyses have, for example, been proposed to examine how often estimates might have the wrong direction or substantially exaggerate an assumed underlying effect.

The broader principle is straightforward: design the study around the distinctions you need the evidence to make.

An informative result depends on the question being asked

There is no context-free definition of an informative effect size. A one-point difference might be negligible in one setting and consequential in another. Likewise, a result can be theoretically informative without immediately changing practice.

Consider three different questions about the same intervention. A researcher interested in whether a psychological mechanism exists may care about evidence that discriminates between competing explanations. A school administrator deciding whether to purchase the intervention may care about whether the improvement justifies its cost. A meta-analyst may value a precise estimate even when the estimated effect is small.

Before designing the study, identify what kind of uncertainty you are trying to reduce and for whom that reduction matters. Later questions about whether the evidence should change what researchers believe or change what someone should do then become much easier to answer.

The best planning question is often counterfactual

Try completing this sentence several times:

“If the study produces ________, I would conclude ________ because ________.”

Fill the first blank with different plausible outcomes. Do not restrict yourself to the result your hypothesis predicts.

If every version leads to essentially the same conclusion, your design may not be capable of distinguishing the explanations you care about. If several plausible outcomes lead only to “we need another study,” consider whether the current design will narrow the important uncertainty enough to matter.

That does not mean every possible outcome must yield a decisive answer. Sampling variability, measurement limitations, model uncertainty, and unexpected data patterns make that unrealistic. The objective is to identify whether the study has a reasonable prospect of producing evidence that discriminates among the possibilities that motivated it.

Planning what different results would mean

Notice that Result B may be more informative than Result C even though its estimated benefit is smaller. Informativeness depends partly on what the evidence allows you to distinguish, not simply on whether the estimate points in the desired direction.

This exercise can reveal design problems before they become expensive. If the planned sample, measurements, or analysis would routinely produce intervals wide enough to span both negligible and consequential effects, you may need to reconsider the design before collecting data.

Common ways researchers mistake a result for an informative one

Reverse-engineer the study from the conclusions you need

Before finalizing your design, write down several plausible results and the conclusion you would draw from each. Include at least the outcome predicted by your hypothesis, a substantively small result, a result near zero, an unexpected direction, and a result with substantial uncertainty when these possibilities make sense for your question.

Then ask whether your planned design could reliably tell those possibilities apart.

This last question is particularly demanding, but useful. A study does not need to transform a field or immediately alter policy to be worthwhile. It might clarify an estimate, challenge an assumption, establish feasibility, or help determine what research should happen next. The contribution simply needs to be identifiable.

And if you discover that a technically flawless execution would still leave the central question untouched, that is not primarily a statistical problem. It suggests that you should reconsider whether the study is positioned to answer something important in the first place.

Before collecting data, test whether your results could be informative

Questions about planning for informative research results

Design the study around what you need to learn

How to Cite This Guide

This guide is intended to be read, shared, and used in research, teaching, and academic work. If you draw on its ideas, explanations, or other content, please acknowledge the source by citing the guide. Doing so gives appropriate credit and helps your readers locate the original resource.