How Do You Know Whether the Review Included the Right Studies?

How Can You Tell Whether the Right Studies Made It Into a Systematic Review?

A systematic review may search thousands of records and eventually include only a small fraction of them. That reduction is not inherently a problem. Most retrieved records are often irrelevant. The important question is whether the studies that survived the selection process are the ones that genuinely belong in the review.

This requires looking beyond the number of included studies. You need to examine what the review defined as eligible, how those criteria were applied, why apparently relevant studies were excluded, and whether the final evidence base actually corresponds to the question the review claims to answer.

The Right Studies Are the Studies That Fit a Defensible, Prespecified Question

Study Selection Determines What Evidence the Review Actually Represents

Searching and Selecting Studies Are Different Problems

A search determines which potentially relevant records reviewers discover. Eligibility criteria and screening determine which studies ultimately enter the review. A review can perform an excellent search and still construct a problematic evidence base if inappropriate selection decisions follow.

This is why evaluating whether the search was comprehensive enough answers only part of the appraisal question. Once evidence has been found, you still need to determine whether it was selected appropriately.

The Eligibility Criteria Should Follow From the Review Question

Before looking at individual included studies, examine the review's eligibility criteria. These rules define the evidence the review intends to consider.

Depending on the question, criteria may specify populations, interventions or exposures, comparators, outcomes, study designs, settings, follow-up periods, dates, publication characteristics, or other features. The precise framework varies by review type, so not every review should be forced into the same eligibility template.

The important test is conceptual alignment. If a review asks about one population but admits substantially different populations without justification, or claims to evaluate an intervention while including materially different interventions, the synthesis may no longer answer the question readers think it answers.

Broad Eligibility Is Not Automatically Bad

Some reviews intentionally ask broad questions. They may include different populations, settings, intervention variants, or study designs because variation itself is scientifically relevant.

Broad eligibility therefore should not be criticized merely because the included studies differ. The question is whether the breadth was planned, justified, and handled appropriately in the analysis and interpretation.

Problems arise when a review quietly moves from a broad collection of evidence to a narrow conclusion that the assembled studies cannot support.

Narrow Eligibility Can Also Distort the Evidence

Restrictive eligibility criteria may make studies more comparable, but excessive restriction can remove evidence relevant to the stated question. Language, publication status, date, setting, study design, or outcome restrictions can all shape the evidence base.

Some restrictions are entirely defensible. Others may reflect convenience rather than the scientific question. You therefore need to ask not simply whether a restriction exists, but what it excludes and why.

Selection Decisions Require Judgment

Determining whether a study meets eligibility criteria is not always mechanical. Reports may describe populations incompletely, use unexpected terminology, combine eligible and ineligible participants, or leave methodological details unclear.

Cochrane therefore treats study-inclusion decisions as influential judgments and requires at least two people working independently to make final eligibility determinations for potentially eligible studies, with a prespecified process for resolving disagreements. Independent assessment reduces both mistakes and the possibility that one person's expectations shape which evidence enters the review.

Full-Text Exclusions Deserve Attention

The most revealing selection decisions often occur after reviewers retrieve full-text reports. At that stage, apparently relevant studies are excluded because they fail one or more eligibility criteria.

A transparent review should make the selection process understandable and report reasons for consequential exclusions. If a well-known or seemingly eligible study is absent, look for it among excluded studies before assuming the reviewers simply missed it.

Then evaluate the reason. An exclusion may be entirely correct, or it may expose an eligibility rule or interpretation that materially shapes the review's conclusion.

One Study Can Have Several Reports

Another easily missed problem is confusing publications with studies. A single study may generate a protocol, conference abstract, primary article, follow-up report, secondary analysis, and several outcome papers.

Cochrane explicitly treats the study, rather than each report, as the unit of interest. Multiple reports should be linked so that the same participants are not inadvertently counted as separate studies. Duplicate inclusion can bias a meta-analysis by giving one underlying study more influence than it deserves.

The Included Studies Must Fit the Particular Synthesis Too

A study can legitimately belong in a systematic review without belonging in every analysis within that review. Perhaps it measures a different outcome, uses an incompatible follow-up period, or evaluates a different comparison.

Eligibility for the overall review and eligibility for a particular meta-analysis are therefore distinct questions. Before interpreting a pooled estimate, identify exactly which subset of studies contributed to it.

Relevance Does Not Mean Methodological Quality

A study may perfectly satisfy the eligibility criteria and still have serious risk-of-bias concerns. Conversely, a rigorous study may be correctly excluded because it addresses the wrong population or comparison.

This distinction matters. Study selection asks whether evidence belongs in the review. Risk-of-bias assessment asks how much confidence you should place in results from eligible studies. A review needs both.

The Final Evidence Base Should Still Resemble the Original Question

After examining the eligibility rules, look at the characteristics of the included studies as a group. Do they actually represent the population, intervention or exposure, comparator, outcomes, settings, and study designs implied by the review question?

If the answer is only partly yes, determine whether the review acknowledges that limitation. A systematic selection process can still produce a misleading synthesis when the evidence ultimately available is narrower, more indirect, or substantially different from the claim being made. That is one reason a systematic review can remain systematic while still misleading readers.

When the Included Studies Do Not Quite Match the Question

Common Mistakes When Judging Included Studies

Trace the Evidence From the Question to the Final Included Studies

You do not need to repeat the entire screening process to identify major selection problems. Start with the review question, translate it into the main eligibility dimensions, and compare those dimensions with the characteristics of the included studies.

Selection is therefore not about finding the largest possible evidence base. It is about constructing the evidence base that appropriately answers the review question and making the judgments used to construct it visible enough for readers to evaluate.

How to Check Whether the Right Studies Were Included

Questions About Selecting Studies for a Systematic Review

The Right Evidence Base Is Defined by the Question, Not by Its Size

How to Cite This Guide

This guide is intended to be read, shared, and used in research, teaching, and academic work. If you draw on its ideas, explanations, or other content, please acknowledge the source by citing the guide. Doing so gives appropriate credit and helps your readers locate the original resource.