03 · What You Need to Know
The Claim Determines How Strong the Search Needs to Be
There Is No Single Boundary Between Informal and Formal Searching
Literature searches exist on a continuum. At one end, you might type a few terms into a discovery tool, browse promising results, and follow interesting citations. At the other, you might develop database-specific strategies, search multiple information sources, document exact queries, use supplementary search methods, and preserve the process for reproducibility.
Neither end is automatically appropriate or inappropriate. The correct level depends on the purpose.
A quick search may answer "What terminology does this field use?" extremely well. The same search is poorly equipped to answer "Have all relevant randomized trials been identified?" Search quality is therefore better understood as fitness for purpose than as a universal level of procedural complexity.
Positive Existence Claims Need Less Search Certainty Than Absence Claims
Some claims can be established with very little searching. If you locate one genuine study examining a phenomenon, you have evidence that at least one such study exists. Finding additional studies may improve your understanding, but it is unnecessary for that narrow existence claim.
The logic changes when you move from presence to absence. Failing to find a study does not establish that one does not exist. The study may use different terminology, appear in another database, belong to another discipline, or be difficult to retrieve for other reasons.
Presence claim
"At least one study has examined this question." One verified qualifying study can establish the narrow claim.
Distribution claim
"Most studies use this approach" or "research generally finds X." You need enough coverage to characterize the broader literature.
Absence claim
"No studies have examined this question." The search must be capable of finding plausible counterexamples before their apparent absence becomes meaningful.
This asymmetry explains why searching before claiming that no studies exist demands much more than an ordinary literature lookup.
Claims About Frequency Need Representative Coverage
Researchers frequently use phrases such as "most studies," "the literature predominantly," "research typically," or "few studies." These statements sound less absolute than "no studies exist," but they still describe the distribution of characteristics across a body of literature.
If your search preferentially retrieves highly cited studies, papers from one database, recent publications, one discipline, or research using your preferred terminology, you may not have a suitable basis for estimating what is typical or uncommon.
The broader the population of studies implied by your sentence, the stronger the need for a search capable of representing that population.
Claims About Consensus Require Searching for Disagreement
Saying that "research shows" something can imply a level of consistency that may not exist. A few supporting papers cannot establish consensus if contradictory or qualifying evidence was never sought.
Before characterizing a literature as consistent, search deliberately for disagreement. Look for null findings, contrary results, replication attempts, methodological critiques, relevant reviews, and plausible competing interpretations.
An informal search becomes particularly vulnerable when it evolves around a conclusion you already expect. Search methods that repeatedly follow supportive papers through their reference networks can produce a coherent literature simply because the search itself remained within one intellectual neighborhood.
Research-Gap Claims Require the Search to Test the Gap
A gap is not established merely because the first few searches produce little directly matching your question. Low retrieval can result from terminology, indexing, disciplinary boundaries, or an overly narrow conceptualization.
Before making a meaningful gap claim, search alternative terminology, relevant databases, adjacent disciplines, recent reviews, and citation pathways. More importantly, look specifically for evidence that would eliminate or narrow the proposed gap.
The search should be capable of changing your argument. If it is designed only to collect evidence supporting a gap you have already decided exists, it is not seriously testing the claim.
This is why a stronger standard applies before claiming a research gap.
Methodological Claims Can Also Outgrow an Informal Search
The problem is not limited to novelty claims. Statements such as "previous studies have relied mainly on cross-sectional designs," "existing research rarely controls for X," or "most studies use self-report measures" also characterize a body of research.
If these statements are central to your rationale, the search should provide enough coverage to make them defensible. Finding four cross-sectional studies does not demonstrate that longitudinal research is rare unless you searched in a way capable of finding longitudinal studies too.
The Consequence of Being Wrong Matters
Two identical literature claims can warrant different levels of searching depending on what depends on them.
A tentative observation in an early research meeting has limited consequences. The same assertion used to justify a funded trial, expose participants to research procedures, support a clinical recommendation, define a dissertation's novelty, or determine the eligibility of evidence in a systematic review carries considerably more weight.
As the consequences of missing relevant evidence increase, so does the justification for broader databases, better-developed terminology, supplementary searching, stronger documentation, and specialist input.
Informal Searching Becomes Too Weak When Retrieval Itself Becomes Part of the Method
In a systematic review, the search determines which records have an opportunity to enter screening and synthesis. The retrieval process is therefore part of the research methodology.
PRISMA-S was developed specifically because literature-search methods in systematic reviews need sufficiently complete reporting to make the information-retrieval process transparent and reproducible. It contains 16 reporting items covering information sources and search methods. Current Cochrane guidance likewise describes systematic reviews as requiring a systematic and comprehensive approach to identifying studies meeting eligibility criteria.
Once evidence identification itself becomes part of the method, casual browsing is no longer an adequate primary retrieval strategy. A systematic review search requires a different methodological standard.
Informal Does Not Mean Unrecorded
Even when full reproducibility is unnecessary, keeping basic search notes can be valuable. Record useful databases, promising terms, important papers, citation pathways, and the date of searches that may need updating.
This creates continuity between exploration and more rigorous searching. If the project later develops stronger claims, you already have a record of what has been tried rather than attempting to reconstruct several months of browsing from memory and a collection of mysteriously named PDF files.
Watch for Signs That Your Search Has Outgrown Its Original Purpose
A search often begins informally and becomes inadequate gradually. Perhaps you initially wanted background information but now intend to claim that an approach is rarely used. Perhaps an exploratory search becomes the basis for a grant's novelty argument. Perhaps a narrative review begins making conclusions about the entire evidence base.
At each transition, ask whether the search method has kept pace with the claim.
Watch Out
Do not assume that extensive searching is automatically rigorous searching. Spending fifty hours following references informally can still leave systematic blind spots if the same terminology, databases, disciplines, or citation networks dominate throughout.
Stronger Searching Does Not Always Mean a Full Systematic Review
Recognizing that an informal search is too weak does not mean every research question must immediately become a systematic review. There are intermediate levels of rigor.
You can develop a structured search across several appropriate databases, deliberately test alternative terminology, consult relevant reviews, use backward and forward citation searching, document important search decisions, and update the search without claiming systematic-review methodology.
The search should become only as formal as the research purpose requires. Methodological escalation is useful when it reduces a meaningful risk, not because complexity itself is virtuous.
Reproducibility Becomes Important When Readers Need to Inspect Retrieval
If your conclusion depends heavily on which studies were or were not found, readers may need to understand how retrieval occurred. This is particularly true for systematic evidence synthesis and strong negative claims.
At that point, consider whether a reproducible search is necessary. Reproducibility does not guarantee that the search is comprehensive or correct, but it makes the retrieval process inspectable.
A Useful Test: What Paper Would Change Your Sentence?
Imagine that tomorrow someone sends you one relevant paper you missed. Would your statement still be accurate?
If you wrote "several studies have examined X," another study probably changes little. If you wrote "no studies have examined X," one counterexample destroys the claim. If you wrote "research consistently demonstrates X," a well-designed contradictory study may require substantial qualification.
The more vulnerable your sentence is to missing evidence, the stronger the search behind it should be.
