Should You Define What Counts as Relevant Before Seeing the Search Results?

When Should You Decide What Counts as Relevant?

A literature search rarely returns only the studies you need. It may retrieve hundreds or thousands of records ranging from obviously relevant papers to studies that share only a keyword with your question.

The difficult cases sit between those extremes. A paper may examine the right population but a different phenomenon. Another may address your topic in the wrong setting. A particularly interesting study may be only loosely connected to the question but difficult to ignore once you have read it.

If you decide what “relevant” means only after seeing these papers, the results themselves can begin shaping the rules by which they are judged. For structured evidence synthesis, that is a methodological problem. Even for less formal literature reviews, it can make the resulting body of literature difficult to justify consistently.

Define Relevance Before the Results Start Influencing It

Relevance Is a Methodological Decision, Not a Feeling

A paper can be interesting without being relevant

Researchers often use “relevant” informally to mean interesting, related, recent, influential, or useful. Those qualities are not interchangeable with eligibility for a particular research question.

In a structured review, relevance needs an operational meaning. A study belongs because it satisfies criteria derived from what the review is trying to answer. Cochrane describes systematic reviews as bringing together evidence that fits prespecified eligibility criteria, and its current guidance requires review authors to declare the criteria used to determine which studies are included.

PRISMA 2020 similarly asks systematic review authors to report their inclusion and exclusion criteria and explain how studies were grouped for synthesis. That reporting requirement is important partly because readers need to understand the boundaries of the evidence being presented.

Start with the question, not with the papers

A well-formulated review question should guide subsequent methodological decisions, including eligibility criteria, searching, data collection, synthesis, and presentation.

This is why writing down what you are looking for before opening a database is useful. Once the purpose is explicit, you can ask a more demanding question: what characteristics would a study need to have to contribute evidence to that purpose?

The answer will depend on the type of review. Relevant characteristics might concern the population, phenomenon or intervention, comparator, setting, outcomes, time period, study design, publication characteristics, or other features justified by the question. JBI guidance likewise emphasizes that inclusion criteria should be clear and aligned with the review objective and question.

Predefining relevance reduces result-driven decisions

Imagine encountering a large, well-written study from a prestigious journal that is adjacent to your question but does not actually meet the scope you intended. Once you have seen it, there is a temptation to reinterpret the scope so that the paper fits.

The opposite can happen too. A study whose findings complicate your emerging interpretation may suddenly seem “not quite relevant.”

Prespecification does not eliminate judgment, but it constrains that judgment. Instead of asking, “Do I want this paper in my review?” you can ask, “Does this paper satisfy the criteria that follow from my question?”

Relevance and searchability are not identical

A criterion can matter during screening without necessarily belonging in the database search.

For example, you may care about a particular outcome but decide not to require outcome terminology in the search strategy because outcomes can be inconsistently reported in titles, abstracts, and indexing. Cochrane guidance specifically notes that studies generally should not be excluded merely because an outcome of interest is not reported, since non-reporting does not establish that the outcome was never assessed. There are situations where outcome-based eligibility is justified, but the rationale should be explicit.

This produces an important separation:

Your search can therefore be intentionally broader than your final included evidence. The search aims to avoid missing potentially eligible studies; screening then applies the relevance criteria more precisely.

Define criteria clearly enough that another person could apply them

“Studies related to student learning” is not particularly operational. Does perceived usefulness count? What about satisfaction, grades, retention, engagement, self-regulation, or writing quality?

The more consequential the review, the more useful it becomes to express criteria in terms that can be applied consistently. Cochrane recommends pilot-testing eligibility criteria on a small set of clearly eligible, clearly ineligible, and doubtful reports. This can expose ambiguous wording before large-scale screening begins.

This is especially valuable when several reviewers will screen records. Cochrane's methodological standards require at least two people working independently for final eligibility decisions in intervention reviews, partly to reduce errors and the influence of an individual reviewer's biases.

Predefined does not mean unchangeable

No eligibility framework anticipates every awkward paper. You may encounter mixed populations, unconventional study designs, ambiguous interventions, overlapping reports, or studies that sit precisely on a boundary you had not considered.

Methodological rigor does not require pretending these cases do not exist. Cochrane permits sensible post hoc decisions when unexpected issues arise, but changes should remain faithful to the review objectives, be documented, and not be driven by the study findings or synthesis results.

That distinction matters. Clarifying an ambiguous criterion because it cannot be applied consistently is different from changing a criterion because you have discovered a study you want to include.

What Happens When Relevance Is Defined Before Screening?

The criteria do not make every decision automatic. They make the basis of the decision visible and contestable, which is considerably more useful than relying on an undefined sense that a paper “looks relevant.”

Common Mistakes When Deciding What Counts as Relevant

Turn “Relevant” Into Criteria You Can Defend

Before substantial screening begins, write down what would make a retrieved study belong in your evidence base. Ask what must be true about the study for it to contribute to your question.

Then test those rules against a few examples. If two reasonable reviewers could interpret a criterion very differently, clarify it before screening hundreds of records.

The next decisions often concern which characteristics should define that evidence base, including which study designs matter, which populations belong, and which outcomes matter to the question.

Before You Screen the Results

Questions About Relevance and Eligibility

Decide What Belongs Before the Results Become Persuasive

How to Cite This Guide

This guide is intended to be read, shared, and used in research, teaching, and academic work. If you draw on its ideas, explanations, or other content, please acknowledge the source by citing the guide. Doing so gives appropriate credit and helps your readers locate the original resource.