03 · What You Need to Know
How to Choose and Evaluate the Comparator for an Index Test
First, Distinguish the Index Test From the Reference Standard
The index test is the diagnostic procedure whose performance researchers want to evaluate. The reference standard is the method used to establish the target condition's status for comparison.
Suppose researchers are evaluating a new imaging algorithm intended to identify a particular pathological condition. The algorithm is the index test. An appropriate pathology assessment, where available and suitable, might serve as the reference standard.
The index test produces a classification, while the reference standard provides the classification against which that result is evaluated.
This distinction does not imply that the reference standard is infallible. It means the standard is sufficiently credible for the study's intended diagnostic question.
The methodological challenge is to identify the most appropriate reference standard when perfect classification may not be possible.
Why Is the Most Commonly Used Diagnostic Test Not Automatically the Best Comparator?
A test may be widely used because it is inexpensive, accessible, rapid, or familiar to clinicians. Those advantages do not necessarily make it the most accurate method for establishing disease status.
For example, a brief clinical assessment may be the usual first step in diagnosing a condition, but a more comprehensive evaluation may provide a stronger basis for determining whether the condition is present.
If researchers compare a new test only against the routine assessment, they may be measuring agreement with current practice rather than accuracy against the target condition.
This can be useful when agreement is the actual research objective. However, it should not automatically be presented as diagnostic accuracy.
Researchers should distinguish the best available disease-status determination from the test that is most convenient to administer.
What Is the Difference Between a Reference Standard and a Competing Index Test?
Sometimes researchers want to know whether a new test performs better than an existing test. In that situation, the existing test may be a competing index test rather than the reference standard.
Both tests can be evaluated against the same reference standard to compare their diagnostic performance.
| Component |
Primary Role |
Example |
| New index test |
Diagnostic procedure being evaluated |
A newly developed diagnostic questionnaire |
| Existing competing test |
Alternative diagnostic procedure whose performance may also be evaluated |
A commonly used screening questionnaire |
| Reference standard |
Determines the target condition's status for accuracy assessment |
A suitably validated comprehensive diagnostic assessment |
When both index tests are administered to the same participants, their sensitivity and specificity can be compared using paired-data methods that account for the correlation between their results.
However, a shared reference standard must still be appropriate. Comparing two tests against the same flawed standard does not eliminate reference-standard error.
How Does the Target Condition Determine the Reference Standard?
The reference standard must correspond to the condition that the index test claims to identify.
A test intended to detect a current infection requires a reference procedure appropriate for establishing current infection. A test intended to identify structural abnormalities may require a different type of assessment.
Similarly, a questionnaire designed to identify a psychiatric disorder should be evaluated against an appropriate diagnostic assessment rather than merely another symptom questionnaire, unless that comparison is specifically justified.
Researchers should define the target condition operationally, including relevant diagnostic criteria and any severity or subtype distinctions.
If the reference standard classifies a broader or narrower condition than the index test is intended to detect, apparent disagreement may reflect inconsistent definitions rather than poor index-test performance.
What If No Perfect Reference Standard Exists?
Many conditions cannot be established with complete certainty using a single procedure.
Researchers may therefore consider a composite reference standard, expert adjudication, longitudinal follow-up, or statistical approaches designed for situations without a perfect reference standard.
Each option introduces methodological considerations.
A composite reference standard combines information from multiple assessments according to predefined rules. Expert adjudication uses qualified assessors to evaluate available evidence. Longitudinal follow-up may help establish diagnoses that cannot be determined reliably at the initial assessment.
Latent class models may estimate diagnostic properties without treating any observed test as perfectly accurate, but their conclusions depend on assumptions about the relationships among tests and the underlying condition.
None of these approaches automatically eliminates uncertainty. Researchers must explain why the selected method is appropriate and how its limitations affect interpretation.
Why Is Agreement With Another Test Not the Same as Diagnostic Accuracy?
Two diagnostic procedures may produce identical classifications while both misclassifying the target condition.
For example, suppose two questionnaires classify the same participants as positive and negative. Their agreement may be high because they measure similar symptoms or use similar thresholds.
That agreement does not establish whether the participants actually have the condition.
Diagnostic agreement
Describes the extent to which two assessments produce the same or similar results, without necessarily establishing which result is correct.
Diagnostic accuracy
Describes how well the index test identifies the target condition relative to an appropriate reference standard.
Agreement measures such as percentage agreement or Cohen's kappa may be useful for particular questions, but they do not substitute for sensitivity and specificity when the objective is diagnostic classification accuracy.
Why Should Both Tests Be Applied to the Same Participants?
Applying the index test and reference standard to the same participants generally permits direct classification of true positives, false positives, false negatives, and true negatives.
It also reduces concerns arising from differences between separately recruited groups.
However, administering both procedures to everyone may be impractical when the reference standard is invasive, expensive, or ethically difficult to justify.
In such circumstances, researchers may need alternative verification strategies and statistical methods.
The important question is whether participants receive appropriate verification of the target condition and whether the verification process could distort the estimated accuracy.
How Can Selective Verification Distort the Comparison?
Verification bias can arise when the probability of receiving the reference standard depends on the index test result or other factors related to disease status.
For example, if only participants with positive index tests undergo definitive assessment, false-negative cases may remain unidentified.
Calculating sensitivity and specificity using only verified participants may then produce biased estimates.
The direction and magnitude of bias depend on the verification process and the underlying relationships among tests, disease status, and participant characteristics.
Researchers should therefore consider how verification bias arises and how it can be addressed before collecting data.
What Is Incorporation Bias?
Incorporation bias occurs when the index test, or information derived from it, forms part of the reference standard in a way that compromises the independence of the comparison.
Suppose a new questionnaire is being evaluated against a clinical diagnosis, but the clinician uses the questionnaire score as an important criterion for assigning that diagnosis.
The index test is then helping determine the classification against which it is being evaluated.
This can artificially increase apparent agreement and diagnostic accuracy.
Where possible, the reference-standard determination should be made independently of the index test. If incorporation cannot be avoided, researchers should document it and evaluate its implications.
Why Does Blinding Matter When Interpreting the Tests?
Knowledge of the index test result may influence the interpretation of the reference standard, particularly when the reference assessment involves subjective judgment.
Conversely, knowledge of the reference-standard result may influence interpretation of an index test that requires human assessment.
Independent or blinded interpretation can reduce this form of review bias.
Researchers should specify what information interpreters receive and whether that information reflects the test's intended clinical use.
For example, a diagnostic test intended to supplement routine clinical information may legitimately be interpreted alongside certain clinical findings. The design should preserve that intended context without allowing inappropriate knowledge of the reference-standard classification.
Why Does Timing Between Tests Matter?
The index test and reference standard should ordinarily assess sufficiently comparable disease states.
If the target condition changes between assessments, disagreement may arise because the condition genuinely developed, resolved, or changed in severity.
This is especially important for acute infections, fluctuating symptoms, or rapidly progressing conditions.
Researchers should specify the acceptable interval between tests and document treatment or other events occurring between assessments.
The appropriate interval depends on the biological characteristics of the condition and the procedures involved.
How Does the Intended Population Affect Comparator Selection?
The reference standard must be feasible and meaningful for the population in which the index test will be evaluated.
For example, a highly invasive reference procedure may be ethically appropriate for patients already undergoing clinically indicated investigation but inappropriate for healthy volunteers.
Researchers should not solve this problem by recruiting only participants who can conveniently receive the reference standard if doing so substantially changes the population being studied.
Such selection may affect the disease spectrum and applicability of the results.
The need to evaluate tests in the population where they will actually be used should influence both reference-standard selection and recruitment.
What If Different Participants Receive Different Reference Standards?
Sometimes researchers use one reference procedure for some participants and another for others.
This may occur because certain procedures are contraindicated, because clinical pathways differ, or because different levels of diagnostic investigation are available.
However, differential reference standards can create bias if they have different abilities to identify the target condition.
For example, participants with positive index tests may undergo a highly accurate invasive assessment, while those with negative results receive only routine follow-up.
Differences in verification quality can then influence the apparent diagnostic accuracy.
Researchers should justify any differential verification, report the procedures used, and consider suitable sensitivity analyses.
How Should the Comparison Be Reported?
STARD 2015 recommends transparent reporting of the index test, reference standard, participant selection, test execution, interpretation, and flow through the study.
Researchers should describe the rationale for selecting the reference standard and whether interpreters were aware of other test results.
They should report the number of participants receiving each test, missing or indeterminate results, and intervals between assessments.
Accuracy estimates should include appropriate measures of uncertainty and should be interpreted in relation to reference-standard limitations.
QUADAS-2 can help identify risk-of-bias and applicability concerns associated with the index test, reference standard, participant selection, and study flow.
