03 · What You Need to Know
How to Determine Whether an AI-Assisted Search Meets Systematic Review Standards
What Does Reproducibility Mean in Literature Searching?
In literature searching, reproducibility generally concerns whether another researcher can understand and reconstruct the search procedures sufficiently to evaluate how evidence was identified.
A conventional database search can often be described through the database and platform, exact query, field restrictions, filters, search date, and other relevant settings.
Another researcher may then execute the strategy again and examine its retrieval behavior.
However, this does not guarantee identical results. Databases add records, revise metadata, change indexing, and sometimes modify search functionality.
For this reason, reproducibility should not be reduced to obtaining precisely the same number of results at a later date.
The Cochrane Handbook emphasizes documenting search methods in enough detail that searches can be reproduced to the extent possible. PRISMA-S similarly focuses on complete reporting of information sources and search procedures.
These expectations are particularly important because systematic review conclusions depend on the body of evidence made available through the search.
Why Is Saving an AI Prompt Not Enough?
A prompt records what the researcher asked the system to do. It does not necessarily reveal what the system actually did.
Suppose the prompt states: "Find all relevant studies about university faculty trust in generative AI published since 2023."
The AI returns twenty publications.
From the prompt alone, another researcher cannot necessarily determine whether the system searched Scopus, PubMed, a public web index, an internal scholarly collection, or no external source at all.
Nor can the researcher establish whether the system examined twenty records, two hundred records, or a much larger collection before selecting its response.
The model may also generate a different answer when the same prompt is repeated because its configuration, available context, retrieval sources, or response-generation process has changed.
Prompt reproducibility
The ability to preserve and repeat the instruction given to an AI system.
Search reproducibility
The ability to reconstruct and evaluate the actual retrieval procedure, including its sources, search conditions, and handling of records.
Prompt preservation contributes to documenting AI assistance in literature searching, but it cannot substitute for information about the underlying retrieval process.
Which Missing Details Make an AI Search Difficult to Reproduce?
Not all missing information has the same methodological significance.
A system that does not disclose every internal ranking parameter may still provide useful supplementary discovery. A system that cannot identify its searchable sources or retrieved records creates a more fundamental problem when used as the sole evidence-identification method.
| Missing information |
Why it matters |
Potential consequence |
| Searchable sources |
Researchers cannot evaluate which literature was accessible |
Unknown coverage and potentially missing studies |
| Executed queries or retrieval methods |
Search logic cannot be adequately inspected |
Unverifiable retrieval decisions |
| Search dates |
The temporal coverage cannot be established |
Uncertainty about which evidence was available |
| Complete retrieved records |
Researchers cannot determine which candidates were considered |
Unobservable omissions or selection decisions |
| Ranking or selection procedures |
The basis for prioritizing or excluding records may be unclear |
Potential selection bias |
| Model or retrieval configuration |
Later runs may behave differently |
Limited repeatability and difficult troubleshooting |
| Verification and screening procedures |
Researchers cannot establish how candidate studies were evaluated |
Uncertain eligibility decisions |
The importance of each limitation depends on the role assigned to AI. If the system merely suggests synonyms that are subsequently verified and incorporated into a fully documented database search, opaque AI retrieval may not be a central methodological problem.
If the system independently determines which publications enter the review's screening process, the same opacity becomes much more consequential.
What Is the Difference Between an Opaque Search and a Non-Reproducible Search?
Opacity concerns the visibility of a system's operations. Reproducibility concerns whether the relevant procedures can be reconstructed or repeated sufficiently for evaluation.
The concepts overlap but are not identical.
A conventional database may use proprietary ranking algorithms while still allowing researchers to preserve exact Boolean expressions, search dates, and retrieved records.
Similarly, an AI retrieval system might use a complex model but provide an identifiable source collection, recorded queries, exported results, and sufficient documentation of its operation.
Such a system may support a more auditable search than a conversational tool that returns only a short, unexplained list.
However, even detailed reporting does not guarantee comprehensive retrieval. Reproducibility and search sensitivity are separate methodological concerns.
Does PRISMA-S Require AI Searches to Produce Identical Results?
No. PRISMA-S is a reporting guideline, not a requirement that every search produce identical results indefinitely.
Its sixteen items address information sources, search strategies, supplementary methods, and record management.
It recommends providing the complete search strategies for each database and information source as actually executed.
Importantly, PRISMA-S also recognizes that some supplementary search methods involve proprietary or opaque algorithms whose results may not be exactly replicable later.
In these circumstances, the guideline emphasizes reporting the methods used and relevant starting information as transparently as possible.
This distinction matters for AI-assisted searching. A method's limited reproducibility does not automatically prohibit its use as a supplementary discovery technique. However, it does not justify presenting an opaque recommendation list as a comprehensive systematic search.
The PRISMA-S explanation and elaboration provides guidance on reporting these methods.
Can an AI Search Be Reproducible but Still Methodologically Inadequate?
Yes. This is an important distinction that can be overlooked when discussions focus exclusively on transparency.
Imagine an AI system that consistently retrieves the same fifty papers from a documented collection. Its behavior is stable, and the retrieval process can be repeated.
However, the collection excludes several major journals or relevant databases. The search may therefore be reproducible while still failing to identify important eligible studies.
Conversely, a search may retrieve a large number of relevant publications but provide insufficient documentation for another researcher to evaluate its methods.
Reproducibility
Concerns whether the retrieval procedures can be reconstructed and evaluated.
Search adequacy
Concerns whether the sources, strategies, and supplementary methods are appropriate for identifying the evidence required by the review.
A defensible systematic review needs both appropriate evidence identification and sufficient methodological transparency.
When Does AI Retrieval Become Too Selective?
Many conversational AI systems are designed to return a manageable answer rather than every record matching a search.
A researcher may ask for relevant studies and receive ten or twenty recommendations without knowing whether additional records were considered.
That may be acceptable for preliminary reading, but it is problematic when the resulting list is treated as the complete set of candidate studies for a systematic review.
The system may prioritize publications according to semantic similarity, source prominence, or other criteria that are not fully visible.
Consequently, relevant but less prominent studies may be omitted before the research team has an opportunity to screen them.
This differs from a database search in which researchers can generally obtain the retrieved record set and apply their own eligibility criteria.
The distinction between AI paper recommendations and an actual literature search is therefore central to evaluating whether the method is suitable for systematic evidence identification.
Can AI-Assisted Semantic Search Meet Systematic Review Requirements?
Potentially, but its suitability must be demonstrated rather than assumed.
A semantic retrieval system may identify relevant publications that structured keyword searches miss. This can be useful for supplementary discovery, search refinement, or identifying additional candidate studies.
However, the researcher should establish which sources the system searches, whether records can be exported, how retrieval is performed, and whether its limitations can be evaluated.
If semantic ranking produces only a small selected list without exposing the broader candidate set, the method may be difficult to justify as the sole systematic search.
In contrast, a system that supports identifiable retrieval sources, transparent procedures, and exportable records may offer more substantial methodological possibilities.
The potential contribution of semantic discovery beyond keyword matches should therefore be evaluated separately from the adequacy of the overall search strategy.
What If AI Performs Automatic Screening or Exclusion?
Automated screening introduces a different methodological risk from search retrieval.
A system may retrieve a broad set of records but then exclude publications it predicts are irrelevant before human reviewers examine them.
If the exclusion procedure is insufficiently validated, eligible studies may be removed without the research team recognizing the omission.
Researchers should therefore distinguish AI-assisted retrieval from AI-assisted screening and document the methods appropriate to each stage.
Where automated screening is used, the review methodology should address the criteria, validation procedures, human oversight, and handling of uncertain classifications.
A transparent search strategy cannot compensate for an inadequately evaluated exclusion process that follows it.
Can Researchers Improve the Reproducibility of AI-Assisted Searching?
Yes, although the extent of improvement depends on the tool's capabilities.
Researchers can preserve prompts and outputs, record tool information and dates, identify connected sources, and export retrieved records where available.
They can also execute AI-suggested search strategies directly in scholarly databases, preserving the final database-specific expressions and results.
Known relevant studies can be used to investigate whether the search retrieves important evidence. Supplementary citation searching may help identify additional publications.
For systematic reviews, involving an information specialist can improve search development and evaluation.
However, documentation cannot reveal information that the tool does not provide. If the system conceals its sources, candidate set, or retrieval procedures, researchers should acknowledge those limitations rather than claiming full reproducibility.
When Should an AI Search Be Treated Only as Supplementary?
An AI-assisted search is more appropriately treated as supplementary when its retrieval coverage or selection procedures cannot be sufficiently established for the review's primary evidence-identification requirements.
For example, a tool that recommends ten conceptually related publications without revealing its searchable collection may be useful for identifying candidate studies.
It is not, by itself, an adequate basis for claiming that all eligible studies were systematically sought.
Similarly, AI-generated lists of seminal publications may support background reading but should not substitute for a documented evidence search.
Researchers should assign the tool a role consistent with what can actually be verified.
What Should Researchers Do When Their AI Search Cannot Be Reproduced?
First, determine which aspects of the process are missing or opaque.
If the problem concerns unavailable query syntax, consider whether the search can be reconstructed through appropriate bibliographic databases.
If the system exposes only selected recommendations, verify those publications and treat them as supplementary candidates rather than a complete retrieval set.
If the source coverage is unknown, avoid making claims about comprehensiveness.
Where necessary, conduct an additional structured search using identifiable information sources and preserve its methods and results.
The appropriate response is not necessarily to discard every publication identified through AI. It is to ensure that the evidence-identification process supporting the review's conclusions meets the required methodological standard.
