Can a Research Question Produce an Answer That Is Technically Correct but Practically Useless?

What if your research gives the right answer to the wrong practical question?

Suppose a study establishes, with strong evidence, that an intervention changes an outcome. The design is credible, the analysis is appropriate, and the conclusion follows from the data.

Then someone asks the awkward question: Is the difference large enough to matter?

The study cannot answer.

This problem is easy to overlook when research questions are framed around whether an effect, difference, association, or relationship exists. A technically answerable question can produce a technically correct answer while leaving the practical question that motivated the research unresolved.

A correct result is not automatically a useful result

Practical questions usually require more than asking whether an effect exists

Statistical significance and practical significance answer different questions

A hypothesis test may evaluate whether observed data are sufficiently incompatible with a specified null hypothesis under a statistical model. It does not tell you, by itself, whether an effect is large enough to matter in the real situation that motivated the study.

This distinction becomes particularly important with large samples. With sufficient precision, very small departures from a point null hypothesis can be statistically detectable even when the difference has little practical consequence.

Conversely, failing to achieve conventional statistical significance does not establish that an effect is practically negligible. The estimate may simply be too uncertain to distinguish a useful effect from a negligible one.

Neither should be substituted for the other.

The research question determines what kind of answer you can obtain

Compare these questions:

Does the new instructional intervention improve examination scores?

Does the new instructional intervention improve examination scores enough to justify its additional cost and instructional time?

The first asks whether there is evidence of an improvement. The second introduces a consequential threshold and a decision context. Answering it may require effect estimates, uncertainty, costs, implementation burden, possible harms, and perhaps outcomes beyond examination scores.

A study designed only for the first question may therefore produce an impeccable answer while providing insufficient evidence for the second.

“How much?” is often more useful than “Is there?”

For many practical decisions, effect magnitude matters more than the bare existence of an effect.

An intervention that improves a 100-point outcome by 0.2 points and one that improves it by 12 points both have effects in the same direction. Treating them as equivalent because both differ statistically from zero discards the information a decision-maker may need most.

This is why researchers increasingly emphasize effect estimation and contextually meaningful thresholds alongside hypothesis testing. Depending on the field, such a threshold may be described using concepts such as a smallest effect size of interest, minimal important difference, or another substantively justified target difference.

These concepts are not interchangeable in every discipline, and there is no universal numerical threshold for a meaningful effect. The criterion should arise from the substantive question rather than from a generic convention.

Meaningful thresholds should be justified before the result is known

Suppose researchers observe an improvement of 2.4 points and only afterward declare that two points constitutes a meaningful improvement. That interpretation is vulnerable to being shaped by the observed result.

A stronger approach is to ask during study planning what magnitude would change the relevant conclusion or decision and why.

Possible justifications include consequences for users, theoretical predictions, costs and benefits, prior evidence, domain-specific standards, or consultation with relevant practitioners and stakeholders. The appropriate justification depends on what the study is trying to accomplish.

Recent methodological work on sample-size planning likewise emphasizes choosing effect-size targets that correspond to the inferential or practical purpose of the study rather than automatically adopting generic effect-size conventions.

Precision can determine whether a practically useful conclusion is possible

An estimate of a five-point improvement is not enough to make a decision if the uncertainty surrounding it ranges from a negligible benefit to a very large one.

Suppose a decision-maker would adopt an intervention only if the true improvement is likely to exceed three points. An estimate of five points with a very wide confidence interval may leave that question unresolved. A similarly sized estimate with much greater precision could support a substantially clearer conclusion.

Practical usefulness therefore depends not only on the point estimate but also on whether the study can narrow uncertainty enough to matter.

Practical usefulness depends on consequences, not effect size alone

Even a very small effect can matter when it affects many people, accumulates over time, concerns a severe outcome, or can be achieved at negligible cost. A larger effect might be unattractive if obtaining it requires substantial expense, creates harms, or displaces a better alternative.

This is why generic labels such as “small,” “medium,” and “large” cannot by themselves establish practical importance. Effect magnitude needs context.

For applied questions, that context may include implementation costs, opportunity costs, adverse effects, scalability, feasibility, equity implications, or the performance of available alternatives. Not every study must measure all of these. But the question should be framed so that the evidence produced is relevant to the decision the research is supposed to inform.

A technically correct answer may also be too abstract for its intended use

Practical uselessness does not arise only from tiny effect sizes.

A study can estimate an average treatment effect accurately even though practitioners need to know whether the intervention works in a particular population. It can show that two variables are associated even though the decision requires knowing whether changing one would alter the other. It can demonstrate predictive accuracy in a controlled dataset while leaving implementation performance unknown.

In each case, the reported answer may be correct on its own terms. The mismatch lies between the question answered and the question someone needs answered.

When a statistically convincing answer does not settle the practical question

Nothing about the statistical result needs to be wrong. The mistake would be claiming that evidence of a nonzero benefit automatically establishes that adoption is worthwhile.

Now suppose the researchers had established in advance, using a defensible decision process, that an improvement of at least four points would be needed before adoption became attractive. A design capable of estimating whether the benefit plausibly exceeds that threshold would speak much more directly to the decision.

The relevant threshold could also be much smaller than four points if the intervention were inexpensive, scalable, and consequential across a large population. The number itself is not the lesson. The need to connect the estimate to its context is.

Where technical correctness gets mistaken for practical usefulness

Write the question around the decision you actually need to inform

If practical usefulness is part of your research rationale, do not stop at asking whether something works, predicts, differs, or correlates. Ask what magnitude, precision, comparison, and consequence would make the answer useful.

This does not mean converting every research question into a cost-benefit analysis. It means matching the evidence to the claim.

If you say a study will inform practice, ask whether the evidence could actually change what someone should do. If the purpose is scientific rather than practical, ask whether the findings could change what researchers should believe.

Either can justify a study. Confusing them usually cannot.

Before asking an applied research question, check whether the answer will be usable

Questions about practical significance and useful research answers

Ask for the evidence the real decision requires

How to Cite This Guide

This guide is intended to be read, shared, and used in research, teaching, and academic work. If you draw on its ideas, explanations, or other content, please acknowledge the source by citing the guide. Doing so gives appropriate credit and helps your readers locate the original resource.