What Questions Remain Unanswered Because Researchers Keep Studying the Same Comparison?

What If Researchers Keep Asking the Same Comparative Question?

A new intervention is compared with no intervention. Then another study compares it with no intervention. A third uses the same comparison. Years later, researchers know increasingly precisely that the intervention performs differently from doing nothing.

But nobody has tested it against the alternative people actually use.

This is a distinctive kind of research gap. The literature may be large, methodologically respectable, and internally consistent, yet remain poorly aligned with the decision that practitioners, policymakers, organizations, or other stakeholders face.

The problem is not too little research in general. It is that research effort has become concentrated on a comparison whose informational value may now be limited while other consequential comparisons remain uncertain.

Ask Whether Existing Comparisons Match the Choices People Actually Face

The Comparator Determines the Question the Study Actually Answers

“Does it work?” always contains an implicit comparison

An intervention does not simply “work” in isolation. Effects are defined relative to something.

A new teaching strategy may outperform no additional instruction yet perform similarly to an established teaching strategy. A new technology may improve outcomes compared with usual practice but offer no advantage over a cheaper alternative. A new program may outperform a waiting-list condition while being less effective than another active program.

These are different research questions.

Evidence for the first question cannot automatically answer the second.

The easiest comparator is not always the most informative comparator

Researchers may choose control conditions for many defensible reasons. No-treatment, wait-list, placebo, business-as-usual, or simple comparison groups can help isolate particular effects and may be appropriate for early-stage research.

Yet once a basic effect has been repeatedly demonstrated, continuing to use the same weak or minimally active comparator may yield diminishing informational returns.

The decision facing a practitioner is often not “Should I use this intervention or do absolutely nothing?” It may be “Should I replace what I already use with this alternative?”

The comparison required by the second question is different.

Comparator choice changes the interpretation of the effect

Imagine an intervention improves an outcome by ten points compared with no intervention. That sounds substantial.

Now imagine the established alternative improves the same outcome by nine points compared with no intervention.

The practically relevant difference between the new and established options may be much smaller than the original ten-point effect suggests.

This does not invalidate either study. It demonstrates that effects are comparative quantities. Changing the comparator changes the question and potentially the decision.

A field can become very certain about a low-value comparison

Replication and cumulative evidence are important. Researchers should not abandon a comparison simply because somebody has studied it once.

However, there is a point at which repeated studies may primarily increase precision around a contrast that is already sufficiently understood while more consequential alternatives remain untested.

AHRQ's work on future research needs emphasizes that evidence gaps should be prioritized according to their potential to produce actionable findings and improve decisions, rather than treated as equally valuable merely because they exist. AHRQ also notes that areas already containing research may still require different research approaches to generate more useful evidence.

This provides a useful principle beyond healthcare: research value depends partly on what uncertainty the comparison resolves.

Repeated comparisons can conceal several unanswered questions

The missing comparison should correspond to a real uncertainty

There may be dozens of theoretically possible pairwise comparisons among available interventions. It would be wasteful to study every combination merely because each empty cell can be called a gap.

The stronger question is: which untested comparison would materially change what someone knows or decides?

Perhaps two interventions are both widely used but have never been compared directly. Perhaps one is substantially cheaper but its relative effectiveness remains uncertain. Perhaps a new intervention is routinely compared with no treatment even though no treatment is unrealistic in normal practice.

These situations provide stronger rationales because the missing comparison corresponds to an actual scientific or practical choice.

Indirect comparisons may provide information, but they require assumptions

A direct head-to-head study is not always necessary to learn about relative effects.

If intervention A has been compared with C and intervention B has also been compared with C, researchers may sometimes estimate the relative effects of A and B indirectly. Network meta-analysis extends this principle across networks of interventions and can combine direct and indirect evidence when relevant assumptions are sufficiently plausible.

This means that “A and B have never been directly compared” does not automatically imply that nothing is known about their relative performance.

Before claiming a head-to-head gap, determine whether credible indirect evidence already exists and what uncertainty remains after considering it.

Direct comparisons can still matter when indirect evidence is uncertain

Indirect comparisons depend on the comparability of studies across the evidence network. Differences in populations, outcome definitions, study designs, intervention implementation, or effect modifiers can weaken the inference.

A direct comparison may therefore be valuable when it resolves uncertainty that indirect evidence cannot adequately address.

The justification should identify that uncertainty rather than simply saying the two options have never appeared in the same trial.

Researchers can repeatedly compare variables rather than alternatives

The same-comparison problem is not limited to intervention trials.

A correlational field might repeatedly study the association between technology acceptance and behavioral intention while neglecting actual behavior, learning, persistence, or other outcomes. An educational field might repeatedly compare online and face-to-face instruction while leaving unanswered which design features within either format actually matter.

Here the problem is conceptual repetition. Researchers continue testing a familiar relationship while more informative contrasts remain unexplored.

Sometimes this overlaps with measuring outcomes that do not answer the consequential question. In other cases, the repeated comparison itself is the limitation.

Another comparison is useful only if it changes the information structure

Suppose twenty studies compare method A with method B and estimates remain highly uncertain because all studies are small and biased. The problem may not be that researchers keep studying the same comparison. The problem may instead be weak evidence about an important comparison.

Conversely, if A versus B is already well established but stakeholders must choose between A and C, continuing to refine A versus B may have less informational value.

These are different diagnoses and imply different next studies.

Research prioritization matters because every comparison has an opportunity cost

Time, participants, funding, researcher attention, and publication capacity are finite.

A study answering an already well-resolved comparison consumes resources that cannot simultaneously be used to address another uncertainty. This does not make replication wasteful. Replication can test robustness, reproducibility, transportability, and precision.

The relevant question is whether the expected informational gain justifies another iteration of the same comparison.

The research gap may be comparative rather than absolute

A weak gap statement says:

“No study has compared intervention A and intervention C.”

A stronger formulation explains why that comparison matters:

“Interventions A and C are both feasible options for the same decision, but existing evidence primarily compares each with minimal intervention. Their relative effectiveness therefore remains uncertain, limiting the evidence available for choosing between them.”

The second statement identifies the decision that existing research cannot support.

When Repeatedly Beating “No AI” Stops Answering the Useful Question

The original comparison was not necessarily poor. It may have been exactly what early research needed. Its informational value changes as the evidence base matures.

Common Mistakes When Looking for Comparative Research Gaps

Map the Comparisons Before Adding Another One

When reviewing a literature, do not record only what interventions, variables, or approaches have been studied. Record what they have been compared with.

A simple comparison map can reveal that a supposedly large literature is concentrated around a surprisingly small number of contrasts.

This shifts the literature review from cataloguing studies to evaluating the information architecture of the field. You can see not only what has been investigated, but which choices the evidence can and cannot support.

If the same comparison persists because the available designs cannot address a more ambitious question, you may also need to examine whether the dominant study designs are limiting what can be learned.

Before Proposing a New Comparison, Check Its Informational Value

Questions About Repeated Research Comparisons

A Large Literature Can Still Ask the Same Question Repeatedly

How to Cite This Guide

This guide is intended to be read, shared, and used in research, teaching, and academic work. If you draw on its ideas, explanations, or other content, please acknowledge the source by citing the guide. Doing so gives appropriate credit and helps your readers locate the original resource.