Should Researchers Manually Audit a Sample of AI Decisions in a Systematic Review?

If AI Screens Thousands of Records, How Many Should You Check Yourself?

Suppose generative AI screens 5,000 titles and abstracts for a systematic review. Checking every decision manually may eliminate much of the time the researchers hoped to save. Checking only a handful of records, however, may overlook important errors.

One possible compromise is to manually audit a sample of AI decisions. Researchers could examine selected records, compare the AI classifications with independent human judgments, and determine whether the results raise concerns about the screening workflow.

The challenge is deciding what to sample, how many decisions to verify, and what conclusions can legitimately be drawn from the audit. A sample that contains no errors is not necessarily evidence that the remaining decisions are correct.

Yes, but the Audit Must Be Designed to Detect Consequential Errors

How to Design a Meaningful Audit of AI Screening Decisions

What Is a Manual Audit of AI Decisions?

A manual audit is a structured examination of AI-generated decisions against independently established human judgments or another appropriate reference standard.

In systematic review screening, an audit may involve selecting records classified by AI, examining their titles and abstracts or full texts, and determining whether the AI applied the eligibility criteria correctly.

The purpose is not simply to find examples where AI performed well. It is to identify errors, estimate their frequency where the sampling design permits, and determine whether those errors undermine the intended workflow.

Auditing should be distinguished from routine verification. An audit evaluates the performance of a decision process, while verification may involve checking each consequential decision before it is accepted.

Why Audit AI Exclusions Rather Than Only AI Inclusions?

When AI retains an irrelevant record, the consequence is usually additional assessment work. When AI excludes an eligible record, that study may never reach full-text assessment.

Consequently, auditing AI exclusions is particularly important when the workflow allows automated removal of records.

However, examining exclusions alone does not provide every performance measure. Sensitivity requires knowing how many reference-positive records were retained and how many were missed. A sample drawn only from excluded records may identify false exclusions, but it does not directly establish sensitivity without additional information or an appropriate sampling design.

The distinction matters because researchers sometimes report the proportion of incorrect exclusions as though it were the false-negative rate. These quantities use different denominators.

What Are the Main Sampling Options?

Different sampling strategies answer different questions. The choice should reflect whether researchers want to estimate overall error rates, investigate difficult cases, or verify a particular class of decisions.

These approaches can be combined. For example, researchers might use a random sample to estimate error frequency and a separate targeted sample to investigate known weaknesses.

Results from targeted samples should not be presented as though they were unbiased estimates of overall screening performance.

Should Researchers Audit AI-Retained and AI-Excluded Records Separately?

Stratification can be useful because AI-retained and AI-excluded records may have very different error profiles.

Researchers might randomly sample records from both groups and independently reassess them. This provides information about false exclusions among AI-excluded records and unnecessary retention among AI-retained records.

If the sampling fractions differ, the observed results should be interpreted within their strata or combined using appropriate statistical weights. Simply pooling the audited records can produce a misleading overall error estimate.

Where the principal concern is missing eligible studies, the audit should devote sufficient attention to AI exclusions rather than allowing the sample to be dominated by records already retained for further review.

How Large Should the Audit Sample Be?

There is no universal sample size that makes AI screening reliable. The appropriate number depends on the purpose of the audit, the expected error rate, the desired precision, and the consequences of errors.

For example, auditing 20 exclusions and finding no mistakes provides limited reassurance. Even a relatively high underlying error rate could produce a sample with no observed errors by chance.

Sample-size planning should distinguish between estimating an error rate with a desired confidence interval and testing whether the error rate is below a predefined threshold. These are different statistical objectives.

Researchers should also consider whether the records are sampled independently, whether the sampling is representative, and whether some records have a higher probability of selection.

What Does Zero Errors in a Sample Actually Mean?

Zero observed errors does not prove that the underlying error rate is zero.

For a simple random sample with independent observations and no detected errors, the approximate rule of three provides a useful illustration of the upper 95% confidence bound for the error probability.

The result does not mean that exactly 3% of all AI exclusions are wrong. It means that, under the stated assumptions, the data are compatible with an error probability up to approximately 3% at the corresponding confidence level.

It also does not establish sensitivity. The calculation concerns the error proportion within the sampled population, such as AI-excluded records.

When sampling without replacement from a finite set of records, exact finite-population methods may provide a more appropriate assessment. The rule of three is an approximation, not a universal audit standard.

How Is an Audit Error Rate Different From Screening Sensitivity?

Suppose researchers audit 100 AI-excluded records and find that 4 should have been retained. The observed false-exclusion proportion among the audited exclusions is 4%.

That quantity uses AI-excluded records as its denominator. Screening sensitivity instead uses all reference-positive records as the denominator.

Both are informative, but they answer different questions. Researchers should use the correct denominator and avoid reporting an exclusion-audit percentage as though it measured the proportion of eligible studies missed.

Calculating screening sensitivity and related metrics requires an evaluation design that supports the necessary reference-positive and reference-negative counts.

Should Difficult or Ambiguous Cases Be Audited?

Yes, but their role in the evaluation should be explicit. Records involving mixed populations, complex interventions, unclear study designs, or incomplete abstracts may reveal systematic weaknesses that random sampling alone misses.

For example, AI might consistently exclude quasi-experimental studies because it interprets the review's design criteria too narrowly. A targeted audit could identify this problem even if such records represent a small proportion of the dataset.

However, a targeted sample enriched with difficult cases will not necessarily estimate the overall error rate accurately. Researchers should distinguish diagnostic testing from representative performance estimation.

Can Auditing Only Human-AI Disagreements Detect Every Error?

No. If the human reviewer and AI agree on an incorrect exclusion, a disagreement-only audit will not identify the mistake.

Agreement may be particularly misleading when most retrieved records are irrelevant. Both classifiers may correctly exclude many records while sharing errors on the relatively few eligible studies.

Therefore, an audit should consider agreement cases as well as disagreements when the objective is to evaluate the reliability of a combined human-AI workflow.

This is especially important when researchers are considering AI as an alternative to a second reviewer.

What Should Happen When an Audit Finds Errors?

An audit is useful only if its findings influence the workflow. Researchers should define in advance what will happen when consequential errors are detected.

Possible responses include investigating the affected eligibility criteria, examining similar records, expanding the audit, repeating classifications, or returning to human screening.

When false exclusions are found, researchers should not assume that correcting only the sampled mistakes resolves the problem. Similar errors may exist among unaudited records.

The decision to expand verification should consider the type and frequency of errors, their consequences, and the methodological standards governing the review.

Does PRISMA Require a Particular Audit Percentage?

PRISMA 2020 does not prescribe a universal percentage of AI decisions that must be manually audited.

Its selection-process reporting requirements instead emphasize describing how studies were selected, how many reviewers participated, whether they worked independently, and how automation tools were used.

The PRISMA 2020 explanation and elaboration guidance specifically addresses reporting validation procedures for internally developed automation classifiers and the handling of automated exclusions.

Researchers should therefore report the actual audit design and its limitations rather than claim compliance based on an arbitrary percentage.

Auditing AI Exclusions in an Educational Technology Review

What Does the 3% Result Mean?

In the audited sample, 3% of AI exclusions were incorrect. Because the records were randomly sampled, this provides an estimate of the false-exclusion proportion among the 1,700 AI-excluded records, subject to sampling uncertainty.

It does not mean that AI missed exactly 3% of all eligible studies. That would require a different denominator and additional reference information.

If the observed proportion were applied descriptively to the full exclusion population, it would suggest approximately 51 incorrectly excluded records. This is only an estimate, not a verified count, and it carries uncertainty.

The discovery of six false exclusions should prompt further investigation. The team should not simply correct those six records and assume that all remaining exclusions are safe.

What Would a Defensible Next Step Be?

Suppose the missed records disproportionately involve studies where AI-generated feedback is embedded within a broader learning platform. This suggests that the model may be misinterpreting the intervention criterion.

The team could clarify the operational criterion, reevaluate the screening instructions, and examine other records sharing the same characteristics. Depending on the risk and available resources, it may be necessary to expand verification or manually review all proposed exclusions.

The example illustrates the difference between detecting an error and establishing that the entire workflow is sufficiently reliable. An audit can reveal problems, but the research team must decide how those findings affect the screening procedure.

Common Mistakes When Auditing AI Screening Decisions

Design the Audit Around the Decision It Must Support

Before selecting records, decide what the audit is intended to establish. Estimating an error rate, investigating suspected weaknesses, and determining whether a workflow can safely exclude records are related but different objectives.

A representative sample may support statistical estimation, while targeted examination can identify difficult cases. Neither automatically substitutes for the independent human assessment required by applicable review standards.

Should Audits Be Repeated?

Additional audits may be justified when the model, prompts, eligibility criteria, or screening dataset change materially. A validation result obtained under one configuration should not automatically be assumed to apply after substantial changes.

Researchers may also consider periodic checks during long screening projects, particularly when records are processed in batches or the AI service may change over time.

These are practical quality-control considerations, not universal requirements for a fixed audit schedule.

How Should the Audit Be Reported?

Document the audit population, sampling method, sample size, reference-decision procedure, number and types of errors, statistical estimates, and actions taken in response.

If the audit was stratified, explain how strata were selected and whether estimates were weighted. If targeted cases were examined, distinguish those findings from representative error estimates.

PRISMA 2020 emphasizes transparent reporting of automation used in study selection. The detailed explanation of AI involvement in the review should make clear whether AI decisions were independently verified, sampled for auditing, or accepted without human assessment.

Before Accepting the Results of an AI Screening Audit

Frequently Asked Questions About Auditing AI Screening Decisions

An AI Audit Is Only as Strong as Its Sampling and Verification

How to Cite This Guide

This guide is intended to be read, shared, and used in research, teaching, and academic work. If you draw on its ideas, explanations, or other content, please acknowledge the source by citing the guide. Doing so gives appropriate credit and helps your readers locate the original resource.