Should Everyone Receive Both the Index Test and Reference Standard?

Must Every Participant Undergo Both Tests for Diagnostic Accuracy to Be Valid?

You are evaluating a diagnostic test and need to determine whether its positive and negative results correctly identify a target condition. Ideally, every participant would receive the index test and an appropriate reference standard.

But what if the reference standard involves an invasive procedure, substantial expense, radiation exposure, or an investigation that would not ordinarily be justified for people with negative test results?

Can you verify only participants with positive index tests? Can different participants receive different reference procedures? Or would these decisions make the diagnostic accuracy estimates unreliable?

The answer depends on the verification strategy, the target condition, and whether the resulting data can support unbiased or appropriately adjusted estimates of diagnostic performance.

Complete Verification Is Usually Preferable, but Not Always Feasible

How Reference Verification Affects Diagnostic Accuracy

Why Is Complete Verification the Preferred Starting Point?

Diagnostic accuracy studies compare index test results with the target condition status established by a reference standard.

For a binary test, complete verification allows researchers to classify participants into four categories: true positive, false positive, false negative, and true negative.

These categories form the basis for calculating sensitivity and specificity.

When everyone receives an appropriate reference standard, researchers can identify affected participants even when their index test result is negative.

This is essential because false negatives cannot reliably be identified if negative index test results are routinely accepted as proof that the condition is absent.

Complete verification also simplifies the analysis by reducing dependence on assumptions about disease status among unverified participants.

However, complete verification does not automatically eliminate every source of bias. The reference standard may be imperfect, participants may be unrepresentative, or test interpretation may be influenced by knowledge of other results.

What Happens When Only Positive Index Tests Receive the Reference Standard?

Suppose a new diagnostic questionnaire is used to identify patients who may have a particular condition.

Participants with positive questionnaire results undergo a comprehensive diagnostic assessment. Those with negative results receive no further evaluation.

The researchers can determine how many positive results are confirmed by the reference standard. However, they cannot directly determine how many negative results are false negatives.

If they assume that every unverified negative result is a true negative, they may substantially overestimate sensitivity and distort other accuracy measures.

Alternatively, if they calculate accuracy using only verified participants, the resulting estimates may reflect a selected subset rather than the intended testing population.

The problem is called partial verification bias, sometimes described as workup bias.

Its magnitude and direction depend on how participants are selected for verification and how the resulting data are analyzed.

What Is the Difference Between Complete, Partial, and Differential Verification?

Partial and differential verification can occur together. For example, some participants may receive an invasive reference procedure while others receive clinical follow-up, and a third group may remain entirely unverified.

Researchers should describe these arrangements explicitly rather than reporting only the total number of participants with available reference results.

Why Can Partial Verification Bias Sensitivity and Specificity?

Sensitivity is the proportion of participants with the target condition who receive a positive index test result. Specificity is the proportion without the condition who receive a negative result.

Both measures require knowledge of disease status among participants with positive and negative index test results.

If verification depends on the index test, the verified sample may contain a different distribution of disease status and test results from the original study population.

For example, when positive index tests are preferentially verified, false-negative participants may be underrepresented in the verified dataset.

A naive analysis may then produce an artificially favorable estimate of sensitivity.

Specificity can also be distorted, although the direction of bias depends on the verification process and the relationships among test results, disease status, and other participant characteristics.

Researchers should not assume that verification bias always increases every accuracy measure.

The mechanisms underlying verification bias in diagnostic accuracy studies should inform both the design and analysis.

When Is Universal Reference-Standard Testing Unethical or Impractical?

Some reference procedures carry risks that cannot be justified solely for research purposes.

For example, an invasive biopsy may be clinically indicated for patients with suspicious findings but inappropriate for participants with very low estimated disease probability.

Other procedures may involve substantial radiation exposure, sedation, discomfort, or resource demands.

Even noninvasive reference assessments may be too expensive or time-consuming to administer to every participant in a large study.

Ethical review must consider the anticipated scientific value of complete verification against the risks and burdens imposed on participants.

The methodological preference for complete verification does not override ethical obligations.

Instead, researchers should develop a design that addresses the diagnostic question as reliably as possible within acceptable ethical and practical constraints.

Can Researchers Verify a Random Sample of Index-Negative Participants?

Yes. A planned sampling strategy may allow researchers to obtain reference-standard information from a subset of participants who would otherwise remain unverified.

For example, all index-positive participants might receive reference assessment, while a randomly selected proportion of index-negative participants also undergo verification.

If the verification probabilities are known and the sampling mechanism is properly implemented, statistical methods such as inverse-probability weighting may support estimation of accuracy for the original study population.

However, the validity of these estimates depends on the design and assumptions. Every relevant stratum must have a positive probability of verification for straightforward weighting-based recovery of its contribution.

Random sampling does not automatically eliminate problems caused by nonparticipation, reference-standard errors, or missing results among selected participants.

Researchers should specify the sampling probabilities and analysis methods before data collection.

What Is Inverse-Probability Weighting in Verification Studies?

Inverse-probability weighting is one approach to addressing planned or otherwise appropriately modeled incomplete verification.

Verified participants receive weights related to the inverse of their probability of being verified. Participants with lower verification probabilities represent more individuals from the original study population.

For example, if a randomly selected 20% of index-negative participants receive verification, each verified participant in that stratum may represent five participants under a simple design-based weighting scheme.

Such weighting can recover population-level accuracy estimates when the verification design and relevant assumptions are satisfied.

It does not recover information automatically when verification probabilities are unknown, when important selection factors are unmeasured, or when some groups have no possibility of verification.

Other statistical approaches, including multiple imputation and model-based correction, may be appropriate in certain circumstances, but they require their own assumptions.

Can Clinical Follow-Up Replace an Invasive Reference Standard?

Clinical follow-up may be useful when immediate definitive testing is impractical or inappropriate.

Researchers might use subsequent diagnoses, repeat assessments, clinical events, or documented disease progression to clarify the target condition's status.

However, follow-up is not necessarily equivalent to an immediate definitive reference procedure.

A condition may develop or resolve during follow-up. Treatment may alter its course. Some participants may be lost to follow-up, and the probability of receiving additional assessment may depend on symptoms or index test results.

These factors can produce classification errors or differential verification.

Researchers should therefore justify follow-up duration, diagnostic criteria, and how disease changes or missing follow-up will be handled.

The broader issue of selecting a reference standard when no perfect method exists remains relevant.

Why Can Different Reference Standards Create Additional Bias?

Suppose index-positive participants receive a highly sensitive invasive procedure while index-negative participants receive a less sensitive clinical assessment.

Even if every participant receives some form of verification, the procedures may not classify the target condition equally well.

Participants with negative index tests may be more likely to have undetected disease simply because their reference procedure is less sensitive.

The resulting accuracy estimates may reflect differences between reference standards rather than the intrinsic performance of the index test.

Researchers should evaluate whether alternative reference procedures identify the same target condition using sufficiently comparable criteria.

If differential verification is unavoidable, its potential influence should be examined through appropriate analysis and sensitivity assessments.

Should Index and Reference Tests Be Performed Close Together?

Usually, the tests should be performed within an interval that makes their results clinically comparable.

If the target condition can change rapidly, a substantial delay may produce disagreement because the participant's actual disease status has changed.

For example, an infection may resolve between the index test and reference assessment, or a condition may progress to a more detectable stage.

Intervening treatment can also alter the result of the reference procedure.

Researchers should specify acceptable intervals and document clinically relevant events occurring between assessments.

The appropriate interval depends on the condition, test characteristics, and intended diagnostic pathway.

Does Everyone Need to Receive the Index Test?

In a conventional single-index-test diagnostic accuracy study, the index test should ordinarily be administered to the eligible participants whose diagnostic performance is being evaluated.

However, some comparative or two-phase designs may involve different testing arrangements.

For example, researchers might administer an inexpensive initial test to a large cohort and a more resource-intensive index test to a planned subsample.

Such designs can be scientifically defensible when the sampling and analysis are appropriate.

Nevertheless, missing or selectively administered index tests create additional selection and applicability concerns.

The study should clearly identify the population to which each accuracy estimate applies.

Why Must Verification Be Planned for the Intended Testing Population?

The purpose of diagnostic accuracy research is not merely to estimate performance among participants who happen to receive definitive testing.

The intended population may include patients with mild symptoms, uncertain presentations, or low disease probability.

If those participants are disproportionately excluded from verification, the study may no longer reflect the population in which the test will actually be used.

This can compound verification bias with spectrum-related differences in diagnostic performance.

Researchers should therefore establish the intended testing population first and develop the verification strategy around that population rather than allowing reference-standard availability alone to determine eligibility.

How Should Incomplete Verification Be Reported?

STARD 2015 recommends transparent reporting of participant flow, test results, reference-standard procedures, and missing or indeterminate results.

Researchers should report how many participants were eligible, received the index test, received each reference procedure, and contributed to the accuracy analysis.

Reasons for missing verification should be provided where available.

When correction methods are used, investigators should describe the verification mechanism, statistical assumptions, weighting or imputation procedures, and relevant sensitivity analyses.

QUADAS-2 addresses flow and timing, including whether participants received an appropriate reference standard and whether all participants were included in the analysis.

Transparent reporting does not eliminate bias, but it allows readers to judge whether the conclusions are credible.

Why Verifying Only Positive Results Can Conceal False Negatives

The example shows why a planned verification sample can be more informative than simply treating unverified negative results as disease-free.

It does not mean that random sampling is always feasible or sufficient. The reference procedure must remain appropriate, and the analysis must account for verification probabilities, missing results, and sampling uncertainty.

Common Mistakes in Diagnostic Reference Verification

How to Design a Defensible Verification Strategy

Start by determining whether the reference standard can be administered safely and appropriately to everyone in the intended testing population.

If complete verification is feasible, it will often provide the clearest basis for estimating diagnostic accuracy.

If not, identify the reason for incomplete verification and consider whether planned sampling, alternative reference procedures, follow-up, or suitable statistical methods can address the problem.

The protocol should specify who receives each assessment, the timing between tests, the reasons for any alternative verification procedure, and the analysis of incomplete reference data.

Where a sampling design is used, document selection probabilities and account for them when estimating diagnostic performance and uncertainty.

Verification should also be planned with attention to the population in which the test will actually be used. An efficient verification strategy is of limited value if it produces estimates that cannot be applied to the intended diagnostic setting.

Before Finalizing Reference-Standard Verification Procedures

Frequently Asked Questions About Index Test and Reference Standard Verification

Verification Must Support Accuracy Without Creating Unnecessary Participant Risk

How to Cite This Guide

This guide is intended to be read, shared, and used in research, teaching, and academic work. If you draw on its ideas, explanations, or other content, please acknowledge the source by citing the guide. Doing so gives appropriate credit and helps your readers locate the original resource.