How Should the Literature Change the Comparison You Plan to Make?

Compared With What?

You want to know whether a new teaching strategy improves learning, whether an intervention reduces symptoms, whether one technology produces better outcomes, or whether employees working under one arrangement differ from those working under another.

There is a deceptively simple question underneath each of these studies: compared with what?

A new intervention can look beneficial compared with doing nothing and unremarkable compared with an effective existing alternative. Students using a new learning platform may outperform students receiving no additional support but perform similarly to students receiving conventional tutoring. Employees working remotely may differ from office-based employees, but the interpretation depends on what else differs between those groups.

The literature should therefore help you decide not only what you want to study, but what comparison would produce evidence capable of answering the question you actually care about.

The Comparator Determines What Your Study Can Tell You

Your Comparison Defines the Contrast You Are Estimating

An Effect Exists Relative to a Comparison

Statements such as “the intervention improved performance” can conceal an important part of the research design. Improvement relative to what?

In intervention research, an effect is estimated by contrasting outcomes under different conditions. Cochrane guidance explicitly treats the comparator as one of the central elements of PICO alongside the population, intervention, and outcome. The comparator might be no intervention, usual practice, placebo, another intervention, or some other relevant condition.

The distinction matters because different comparisons answer different questions.

The literature can reveal which of these contrasts remains consequential. If ten studies already show that an intervention performs better than no intervention, an eleventh study using the same comparison may add less than a study comparing it with the alternative people actually use.

The Most Convenient Comparator May Not Be the Relevant Alternative

Researchers sometimes select a comparison because it is easy to create. A no-treatment group is simple. A readily available class can become a convenient control. Existing administrative data may divide people into groups that happen to be accessible.

Convenience does not determine interpretability.

Suppose a new instructional platform is intended to replace an existing learning management tool. Comparing the new platform with no digital support might produce a large difference, but that comparison does not answer the decision facing institutions: whether the new platform offers an advantage over the tool already in use.

Similarly, Cochrane notes that trial design can lose clinical applicability when placebo is selected even though an effective intervention is already in regular use. That observation arises from healthcare research, but the underlying reasoning is broader: the relevant comparison depends partly on the real alternative against which the new option will be judged.

“Usual Practice” Needs a Definition

Usual care, standard teaching, business as usual, conventional practice, and similar labels can sound self-explanatory. They rarely are.

What normally happens may vary among institutions, practitioners, classrooms, countries, or time periods. A “usual instruction” comparison could range from a lecture with no feedback to a sophisticated blended-learning environment. Those conditions are not interchangeable merely because both are called conventional instruction.

Cochrane guidance on data collection specifically recommends describing comparison interventions in sufficient detail, including the components of usual care and relevant co-interventions. If the comparison is vague, the estimated contrast becomes vague too.

The Literature May Show That Your Planned Comparison Is Too Easy

A comparison can be technically valid while offering little challenge to the claim being tested.

Imagine evaluating an intensive tutoring program against no additional academic support when effective lower-cost tutoring is already available. Showing superiority to no support does not establish superiority to the realistic alternative.

Or imagine comparing a new assessment method with an obviously weaker procedure even though another well-established method represents current best practice. The resulting study may demonstrate a difference without resolving the decision that motivated the research.

This does not mean an inactive comparator is inherently inappropriate. It may be exactly what the question requires, particularly when you need to estimate the effect of adding something versus not adding it. The literature helps you determine whether that question remains important.

The Literature May Show That You Need More Than One Comparison

Sometimes one comparator cannot distinguish among the explanations you care about.

Suppose students receiving a new collaborative learning activity outperform students who receive no additional activity. Is the difference attributable to collaboration, additional instructional time, increased practice, teacher attention, or some combination?

An active comparison matched on instructional time but lacking the collaborative component could help address a different question. Depending on the design, several comparison groups may allow competing explanations to be examined more directly.

More groups also require more participants, resources, analytical planning, and interpretive care. Add a comparator because it distinguishes a consequential explanation, not because three groups look more sophisticated than two.

Do Not Combine Comparators That Represent Different Questions

The literature may use several comparison conditions that initially look similar enough to treat together. Be cautious.

No intervention, wait-list control, placebo, usual practice, minimal intervention, and active treatment can produce substantially different contrasts. Cochrane guidance therefore emphasizes specifying comparison characteristics and deciding which interventions and comparators can appropriately be grouped for a particular synthesis.

The same principle applies when designing a new study. Ask what exactly the comparison condition represents. If you borrow a comparator from previous research, understand what inferential role it played there before assuming it serves the same purpose in your study.

Observational Comparisons Require a Different Kind of Caution

Not every comparison is experimentally assigned.

You might compare students who use AI frequently with those who use it rarely, employees who work remotely with those who work on site, or patients who receive different treatments in routine practice. These comparisons can be informative, but the groups may differ for reasons other than the exposure or condition being studied.

Those differences create potential confounding. The literature can help identify characteristics associated both with group membership and the outcome, although identifying confounders should follow a defensible causal rationale rather than mechanically adjusting for every variable that previous studies measured.

Changing the comparison may therefore change not only the research question but the assumptions required to interpret the resulting contrast. Those assumptions deserve separate scrutiny when you consider what the literature implies about the assumptions behind your study.

The Comparator Should Match the Decision You Want the Evidence to Inform

One useful test is to imagine that the study is complete.

If the result favors your focal intervention or group, what decision would that result inform? If the answer is “none, because nobody would actually choose the comparator,” reconsider whether you are estimating the most useful contrast.

This is particularly important in applied research. A policymaker deciding between two feasible programs needs evidence comparing those programs, not necessarily evidence comparing one program with nothing. A teacher deciding between two instructional approaches needs a relevant head-to-head contrast. A researcher testing a mechanism may need a carefully constructed control rather than the most realistic real-world alternative.

The appropriate comparator follows the purpose of the inference.

When the Literature Changes the Question From “Does It Work?” to “Is It Better?”

These comparisons answer different questions. AI versus no feedback estimates one contrast. AI versus instructor feedback estimates another. AI plus instructor feedback versus instructor feedback asks whether adding AI produces additional benefit.

The literature has not merely suggested a different control group. It has clarified which contrast is capable of addressing the uncertainty that remains.

Common Mistakes When Choosing a Research Comparison

Choose the Comparison by Working Backward From the Interpretation

Write down the sentence you hope to be able to say after the study. Then ask which comparison would actually justify that sentence.

Then return to the literature and ask what previous comparisons have already established. You may find that repeating the dominant comparison adds little, while a head-to-head or mechanism-focused comparison addresses the more consequential uncertainty.

Also inspect whether previous studies made comparison choices that weakened their conclusions. A poorly specified control, unequal exposure to attention, or an irrelevant comparator may be among the problems you should not repeat.

Will Your Comparison Answer the Question You Actually Care About?

Questions About Choosing a Research Comparison

Your Result Means Something Only Relative to What You Compared

How to Cite This Guide

This guide is intended to be read, shared, and used in research, teaching, and academic work. If you draw on its ideas, explanations, or other content, please acknowledge the source by citing the guide. Doing so gives appropriate credit and helps your readers locate the original resource.