01 · The Question
Would your review look different if the evidence had favored the opposite conclusion?
You begin a literature review with an expectation. Perhaps you believe an intervention works, a theory is persuasive, a technology is harmful, or a particular explanation makes the most sense. That is normal. Researchers do not approach every question as blank cognitive slates.
The problem begins when that expectation influences which evidence gets searched for, retained, trusted, emphasized, or explained away.
Cherry-picking is therefore broader than deliberately hiding an inconvenient paper. It can occur when you search terms associated mainly with one position, stop searching once enough supporting studies appear, scrutinize contradictory studies more harshly, highlight favorable outcomes while ignoring unfavorable ones, or cite repeated claims without tracing the evidence underneath them.
The diagnostic question is uncomfortable but useful: would you apply the same evidential standards if the pattern of findings pointed in the opposite direction?
03 · What You Need to Know
Where can cherry-picking enter a literature review?
Cherry-picking can begin before you read the first paper
A selective literature review does not require consciously deleting inconvenient evidence. The search itself can predetermine what becomes visible.
Suppose you want to investigate harms associated with generative AI. Search terms built entirely around AI harms, AI dependence, academic dishonesty, and negative effects may retrieve a literature already framed around adverse outcomes. A search constructed only around benefits can produce the mirror image.
This is one reason a robust search should represent the phenomenon rather than encode the desired answer whenever possible.
Searching beyond a single set of keywords and, where appropriate, beyond one database can reduce dependence on one vocabulary or disciplinary ecosystem.
Stopping rules can quietly favor the conclusion you wanted
Imagine searching until you find eight studies supporting your expectation and then deciding that you have “enough literature.” If contradictory evidence would have motivated you to continue searching, your stopping decision depends on the results.
Formal evidence syntheses address this problem by prespecifying search methods and eligibility criteria. Less formal literature reviews may not require a protocol, but the underlying principle remains useful: decide what would constitute an adequate search for the question rather than stopping when the literature becomes rhetorically convenient.
The eventual question is whether you have read enough that additional studies are unlikely to materially change your understanding, not whether you have accumulated enough citations to defend your preferred paragraph.
Inclusion criteria should not move after you see the results
Eligibility criteria can become an especially subtle form of selection.
Suppose a study supporting your argument uses a broad age range and you accept it as relevant. A contradictory study uses a similar age range, but you now decide that the sample is “too heterogeneous.” Or you accept self-report outcomes when favorable but criticize the same measurement approach when unfavorable.
Criteria may legitimately evolve during exploratory work when you discover that the original question was poorly specified. But changes should be driven by conceptual or methodological reasons, not by which results survive them.
Watch Out
A criterion that appears only when an inconvenient study arrives deserves inspection. Sometimes it is a valid refinement. Sometimes the literature has simply found the methodological equivalent of a nightclub bouncer.
Critical appraisal can itself become selective
Critical appraisal is supposed to protect against weak evidence. It can also be weaponized against unwanted findings.
A researcher may devote three paragraphs to the limitations of a study contradicting their position while describing a similarly limited supporting study as “compelling.” Small samples become fatal only on one side. Cross-sectional design becomes an important caveat only when the association points the wrong way.
The solution is not to pretend all studies are equally good. It is to apply comparable appraisal standards and explain differences in evidential weight through actual methodological differences.
Evidence weighting
Giving studies different influence because their design, bias, precision, directness, or relevance differs.
Cherry-picking
Giving evidence different treatment because its findings are more or less compatible with the conclusion you prefer.
Unequal weighting is often necessary. Unequal standards are the problem.
Selective outcome attention can change the apparent conclusion
Studies often measure multiple outcomes and time points. An intervention might improve one outcome, have little effect on another, and worsen a third.
If your review reports only the favorable outcome, the study itself becomes selectively represented.
The broader evidence-synthesis literature treats selective non-reporting as a serious source of bias. Cochrane notes that study results may be omitted or incompletely reported because of their magnitude, direction, or statistical significance, causing available evidence to differ systematically from missing evidence. Its ROB-ME framework specifically assesses bias in synthesis arising when studies or particular results are missing because of those result characteristics.
Your literature review can reproduce the same distortion even when the original papers report everything, simply by selecting only the outcomes that support your narrative.
Selective citation can multiply one favorable finding
Suppose one influential study generates four papers. Later reviews cite all four. Your literature review then cites the reviews and concludes that “numerous studies” support the finding.
The apparent evidence has multiplied bibliographically without multiplying empirically.
Check whether you have distinguished multiple papers from independent studies. Otherwise citation volume can become an accidental vote-counting system in which one dataset receives several ballots.
Repeated claims can become detached from their original evidence
Another form of selection occurs through citation chains.
Paper B says Paper A established a claim. Paper C cites B. Paper D cites C. Eventually the claim appears well established because it is repeated across the literature, even though very few authors have examined what Paper A actually demonstrated.
When a claim is central to your interpretation, trace it back to the original evidence. You may discover that the original study was narrower, weaker, or simply different from the proposition later authors repeat.
Contradictory evidence should trigger explanation, not disappearance
If a credible study contradicts your synthesis, you do not need to give it equal weight automatically. You do need to account for it.
Perhaps the study has a serious methodological weakness. Perhaps its population differs. Perhaps it measures a different outcome. Perhaps it reveals a genuine boundary condition.
Those are substantive explanations.
Silence is not.
When important studies conflict, investigate why they disagree rather than treating discordant evidence as an inconvenient footnote.
Publication bias can perform cherry-picking before you arrive
Even a perfectly even-handed reviewer can inherit a selectively visible literature.
Cochrane summarizes substantial evidence that publication and reporting can depend on the magnitude, direction, and statistical significance of results. Studies with statistically significant findings may be more likely to be published, while particular outcomes or analyses may be selectively omitted from reports.
ROB-ME was developed specifically to assess risk of bias in syntheses when entire studies or particular results are missing because of their p-value, magnitude, or direction.
This means avoiding personal cherry-picking is necessary but insufficient. You should also consider whether the available literature has already been selectively filtered before you searched it.
Searches for grey and unpublished evidence can sometimes reveal the missing side
If selective publication is plausible, relevant registries, dissertations, reports, conference materials, or other sources may reveal studies that conventional journal searches miss.
This does not mean grey evidence automatically deserves greater trust. It means that grey and unpublished evidence may need consideration when publication status itself could be related to the findings.
Language can cherry-pick even when the citations are balanced
Suppose your review accurately cites three supportive studies and three contradictory studies. The writing can still be selective.
Supporting findings might be described as “demonstrating,” “confirming,” and “establishing,” while conflicting findings merely “suggest,” “claim,” or “fail to replicate.” Limitations of contradictory studies receive detailed attention while limitations of supportive studies disappear.
Balanced citation counts do not guarantee balanced synthesis.
| Stage |
How cherry-picking can occur |
| Search |
Using terminology or sources that disproportionately retrieve one interpretation |
| Stopping |
Ending the search once enough supporting evidence has accumulated |
| Eligibility |
Applying inclusion or exclusion criteria differently according to study results |
| Appraisal |
Scrutinizing contradictory studies more harshly than supportive ones |
| Outcome selection |
Emphasizing favorable outcomes, analyses, or time points while ignoring relevant unfavorable ones |
| Evidence weighting |
Assigning rhetorical weight according to agreement rather than methodological strength |
| Citation |
Multiplying support through multiple reports or repeated secondary claims |
| Writing |
Using stronger language for preferred findings and skeptical language for conflicting findings without methodological justification |
A disconfirming search is a useful diagnostic habit
After developing an interpretation, deliberately ask what evidence would challenge it.
Search for plausible competing terminology. Examine major papers cited by researchers who disagree. Look at studies using stronger designs that reach different conclusions. Ask what result would make you revise your synthesis and whether such evidence exists.
This is not an instruction to manufacture false balance. A weak contradictory study does not deserve equal weight with a strong body of evidence merely because it is contradictory.
The objective is to give credible disconfirming evidence a fair opportunity to change your mind.
Precommitting to rules reduces flexibility after results are known
Formal systematic reviews use protocols, prespecified eligibility criteria, planned outcomes, and documented synthesis methods partly to reduce decisions that could be influenced by study findings.
Not every literature review requires formal preregistration. But even in narrative work, writing down your scope, inclusion logic, appraisal criteria, and main questions before the synthesis becomes convenient can expose later changes that deserve justification.
The broader principle is simple: the rules used to judge evidence should not depend on whether you like what the evidence says.
04 · A Practical Example
How the same literature can tell two different stories
Hypothetical Example
Does generative AI harm student learning?
Suppose a researcher begins with the expectation that unrestricted generative AI use harms student learning.
The search identifies twelve relevant studies. Four report poorer performance among heavier AI users, three report better performance under structured AI-supported learning activities, three report little clear difference, and two produce mixed results depending on the outcome.
A selective review could cite the four negative studies prominently, dismiss the structured interventions as “not real AI use,” describe the null findings as underpowered, and omit the mixed outcomes from the conclusion.
A defensible review does something harder. It applies the same eligibility rules across studies, evaluates methodological quality independently of direction, distinguishes self-selected AI use from experimentally structured interventions, and asks whether the apparently conflicting results concern the same exposure and outcome.
The final interpretation may still conclude that some forms of AI use are associated with poorer learning. But it will specify which forms, under what evidence, and with what competing findings rather than claiming that the entire literature points one way.
Initial expectation
The researcher begins with a plausible but untested interpretation.
Apply stable criteria
Studies are included and appraised using rules that do not change according to their findings.
Investigate disagreement
Different forms of AI use, study designs, and learning outcomes are separated rather than collapsed.
Weight evidence
Stronger and weaker studies receive different influence for methodological reasons, not because their results are convenient.
Revise the claim
The conclusion becomes more conditional and better aligned with what the entire evidence base supports.
