01 · The Question
Can Learning From Your Search Make the Next Search Biased?
You run an initial literature search and begin reading. The papers teach you terminology you had not considered. You discover alternative spellings, theoretical labels, subject headings, intervention names, and phrases used by researchers in the field. Naturally, you revise your search.
That is often good searching. Search strategies frequently improve through iteration.
But there is another possibility. After reading the first studies, you begin expecting a particular conclusion. New search terms are then added because they retrieve more papers consistent with that conclusion, while terminology associated with competing explanations or contradictory findings receives little attention. The search is still evolving, but it may now be evolving toward confirmation.
03 · What You Need to Know
When Search Refinement Becomes Search Steering
Search strategy development is legitimately iterative
A common misconception is that a rigorous search strategy must spring fully formed from the researcher's mind before the first database query is run. In practice, terminology is often learned through searching.
Cochrane's current guidance explicitly describes search strategy development as an iterative process in which terms are modified based on what has already been retrieved. Researchers may discover relevant text words, controlled vocabulary, alternative terminology, spelling variants, acronyms, or conceptual distinctions only after examining records.
Some review approaches require even more iterative searching. Complex topics and methods such as realist reviews may use multiple search strategies as understanding develops.
The fact that a term was discovered after the first search therefore does not make it methodologically suspect.
The critical question is what the revision is trying to improve
Suppose your initial search concerns whether a particular educational technology affects student engagement. While reading, you discover that relevant researchers frequently use the term “behavioral engagement” rather than simply “engagement.” Adding that terminology may improve retrieval because it better represents the construct you intended to search.
Now suppose the first few studies suggest that the technology improves engagement. You begin adding phrases such as “increased engagement,” “positive engagement,” and “improved participation,” while not exploring terminology associated with disengagement, null effects, adverse outcomes, or competing conceptualizations.
Both cases involve changing search terms after seeing records. Their logic differs.
Search refinement
Terms change to represent the concepts, vocabulary, indexing, or scope of the review more accurately and retrieve relevant evidence more effectively.
Search steering
Terms change partly because the researcher has seen the emerging findings and wants to retrieve more evidence pointing in a preferred direction.
Directional terminology can quietly narrow what you are able to find
Search terms do not merely locate literature. They define part of the route through which literature becomes visible.
If your research question concerns the relationship between an intervention and an outcome, requiring words that imply a favorable result may reduce the opportunity for studies reporting other findings to appear. This can be particularly problematic because titles and abstracts do not necessarily describe results using the terminology you expect.
Cochrane recommends structuring searches around the main concepts of the review and notes that it may be undesirable to search every component of a research question. Outcomes and comparators, for example, may be poorly represented in titles, abstracts, or indexing. Search strategies should generally seek sensitivity while maintaining reasonable precision.
The broader principle is useful beyond intervention reviews: do not make the conclusion you expect a prerequisite for discovering the evidence relevant to the question.
New terminology should ideally broaden conceptual coverage, not merely favorable coverage
Imagine that early reading reveals a new term used by supportive studies. Adding it may be entirely appropriate. But ask whether the same reading process has revealed terminology used in studies that frame the issue differently.
Perhaps researchers use competing theoretical labels. Perhaps different disciplines describe the same phenomenon using different constructs. Perhaps older studies use terminology that has since changed. Perhaps critical studies use a different vocabulary from intervention-oriented research.
If every iteration expands one conceptual branch while leaving alternatives unexplored, the cumulative search can become asymmetric even when each individual addition seems reasonable.
This is where the distinction between searching for evidence and searching for support becomes especially useful. Ask whether each refinement makes the search better at answering the question or merely better at finding more of what you already believe.
Removing terms can create the same problem
Confirmation bias does not require adding favorable terminology. It can also arise when terms are removed.
Suppose a broad synonym retrieves many records that challenge your developing interpretation. You decide that the term is “too broad” and remove it. That may be methodologically justified if the records are genuinely outside scope. But if the term retrieves relevant studies and its main inconvenience is the direction of their findings, the change deserves scrutiny.
A useful test is counterfactual: would you still remove the term if those same records strongly supported your expected conclusion?
If not, the decision may be influenced by the evidence already seen.
Precision is a legitimate concern, but relevance should drive the decision
Search strategies often involve a trade-off between sensitivity and precision. An extremely broad search may retrieve enormous numbers of irrelevant records, while an excessively narrow search may miss relevant evidence.
Cochrane guidance explicitly recognizes this trade-off and notes that search development involves assessing the impact of terms on sensitivity and precision. It also discusses possible stopping considerations, such as whether additional terms retrieve new relevant records or whether removing terms causes relevant records to be missed.
Therefore, removing an unproductive term is not inherently suspicious. The methodological question is whether the records being removed are irrelevant according to the review question or merely inconvenient according to the emerging conclusion.
Known relevant studies can help test a search strategy
One practical way to evaluate revisions is to ask whether the search retrieves studies you already know are relevant, including studies with different findings.
If adding or removing terms causes credible, clearly eligible studies to disappear from the search results, investigate why. The problem may be the search rather than the studies.
This can be particularly revealing when the missing papers are contradictory. A search strategy that reliably retrieves supportive benchmark studies but repeatedly misses known relevant contrary evidence deserves examination.
Changing search terms and changing inclusion criteria are related but different
Search terms determine which records you are likely to encounter. Eligibility criteria determine which retrieved studies enter the review.
Both can become responsive to the findings. You might change terminology so that inconvenient studies are less likely to appear, or change inclusion criteria after seeing which results they retain.
The mechanisms differ, but the underlying danger is similar: the evidence base begins adapting to the conclusion rather than the conclusion adapting to the evidence base.
Watch Out
The fact that a revised search produces “more relevant studies” does not by itself show that it is better. Relevance should be judged against the research question and eligibility criteria, not against whether the new records make the emerging argument more coherent.
Documentation makes iterative searching easier to defend
Cochrane requires the search process to be documented in sufficient detail for correct reporting and, as far as possible, reproducibility. This includes the sources searched, dates, who conducted the searches, and the terms used. PRISMA 2020 similarly asks systematic-review authors to present full search strategies for all databases, registers, and websites searched, including filters and limits.
Keeping versions of important search strategies therefore serves more than administrative neatness. It creates a record of how the search developed.
If a new synonym was added because screening revealed an established term you had overlooked, that rationale can be reconstructed. If a concept was removed because it generated thousands of irrelevant records, the decision can be explained. If terms changed after particular findings appeared, that timing remains visible rather than being rewritten out of the final method.
