Can Different Measurement Devices or Laboratories Introduce Systematic Error?

Can Two Devices or Laboratories Produce Different Results for the Same Quantity?

Your study collects measurements from several laboratories, research sites, or devices. Each uses equipment intended to measure the same quantity, and the resulting values are recorded using the same units.

Yet one laboratory consistently reports higher concentrations than another. Two devices produce slightly different readings for the same participant. A software update changes the values generated by an assessment system.

Are these differences ordinary measurement variation, or could they introduce systematic error into the study?

The concern becomes particularly consequential when different devices or laboratories are associated with particular participant groups, measurement occasions, or research sites. Apparent differences between groups may then reflect the measurement system rather than the underlying phenomenon.

Different Measurement Systems Can Introduce Systematic Error

How Devices and Laboratories Introduce Measurement Differences

What Is Device-Related or Laboratory-Related Systematic Error?

Systematic measurement error is a component of measurement error that remains constant or varies predictably across repeated measurements under specified conditions.

In laboratory and device-based research, systematic error may arise when a measurement system consistently produces readings above or below an appropriate reference value.

For example, a device may consistently report temperatures approximately 0.5 degrees higher than a suitable reference instrument over a particular measurement range.

Another device may exhibit proportional bias, meaning that the difference between its readings and the reference increases as the measured quantity becomes larger.

These patterns differ from random measurement variation, which contributes unpredictable fluctuations across repeated observations.

The distinction between systematic and random measurement error is essential because different error mechanisms require different evaluation and correction procedures.

Why Can Devices Designed to Measure the Same Quantity Produce Different Values?

Measurement systems may differ in their sensing technologies, calibration functions, resolution, detection limits, signal processing, environmental sensitivity, and maintenance history.

Two devices may also use different algorithms to transform raw observations into reported values.

For example, wearable activity monitors may estimate energy expenditure using different combinations of sensor signals and proprietary algorithms. Even when their outputs share the same units, they may not represent identical measurement procedures.

Similarly, automated educational assessment systems may apply different scoring rules or software versions to the same underlying performance data.

Researchers should therefore examine how each device generates its reported measurement rather than assuming equivalence from the device's stated purpose.

What Is Calibration, and Why Does It Matter?

Calibration establishes a relationship between indications produced by a measuring instrument and corresponding quantity values provided by measurement standards, together with their associated uncertainties.

Under the International Vocabulary of Metrology, calibration is not identical to adjustment. Calibration establishes the relationship needed to interpret indications, while adjustment changes the instrument or measurement system.

For example, a weighing instrument may be calibrated using suitable reference masses. The calibration results can identify differences between indicated and reference values across the relevant range.

Those differences may inform correction procedures or decisions about whether the instrument is fit for the intended use.

However, calibration does not guarantee that all subsequent measurements are accurate. Instruments may drift, environmental conditions may change, or the measurement procedure may introduce additional uncertainty.

What Is Measurement Traceability?

Metrological traceability concerns the property of a measurement result whereby it can be related to a reference through a documented, unbroken chain of calibrations, each contributing to measurement uncertainty.

Traceability supports the comparability and interpretation of measurement results across instruments and laboratories.

For example, measurements may be traceable to appropriate reference standards through documented calibration procedures.

However, traceability alone does not establish that a measurement is sufficiently accurate for every research purpose.

The measurement procedure, uncertainty, environmental conditions, and suitability of the reference must also be considered.

Researchers should avoid treating the presence of a calibration certificate as automatic proof that all measurements are interchangeable.

How Can Laboratories Introduce Systematic Differences?

Laboratories may use different equipment, reagents, sample preparation methods, calibration materials, environmental controls, or analytical procedures.

Differences in personnel training, specimen handling, storage duration, and data processing can also influence results.

For example, two laboratories may analyze the same type of biological specimen but use different assay methods. Their reported concentrations may differ systematically even when both laboratories follow their own established procedures.

Such differences do not necessarily establish that one laboratory is performing incorrectly. The methods may have different measurement characteristics or reference systems.

For research comparisons, the important question is whether the results can be interpreted on a sufficiently common measurement basis.

What Is the Difference Between Device Bias and Measurement Variability?

Device bias concerns systematic differences between measurements and an appropriate reference or between methods under a specified comparison framework.

Measurement variability concerns the spread of repeated measurements.

A device may produce highly consistent readings that are systematically too high. Another may produce readings centered near the reference value but with substantial random variation.

These instruments present different measurement problems.

Precision alone does not establish accuracy, and a small average difference does not establish that individual measurements agree sufficiently well.

Researchers should evaluate both systematic differences and random variability in relation to the intended use.

Can Systematic Error Change Across the Measurement Range?

Yes. Some devices exhibit approximately constant bias, while others exhibit proportional or nonlinear differences.

A constant bias means that one device produces readings that differ from a reference by approximately the same amount across the relevant range.

Proportional bias occurs when the difference changes with the magnitude of the measured quantity.

For example, one device may report values approximately 5% higher than a reference across a specified range rather than consistently adding the same numerical amount.

These patterns matter because a single additive correction may not adequately address proportional or nonlinear differences.

Researchers should examine measurement agreement across the range relevant to their study population.

How Can Different Devices Distort Group Comparisons?

Suppose one research site uses Device A and another uses Device B. If the devices differ systematically, observed site differences may partly reflect the measurement systems.

The problem becomes especially serious when device assignment is perfectly aligned with study-group membership.

For example, if all intervention participants are measured with one device and all comparison participants with another, device-related bias may be indistinguishable from a genuine group effect without additional information.

This is a form of measurement error that differs systematically between study groups.

Similar problems can occur in longitudinal studies when the device or laboratory changes between baseline and follow-up.

Researchers should therefore document equipment changes and evaluate whether measurements remain comparable across the study period.

How Should Researchers Compare Two Measurement Devices?

A method-comparison study examines whether two measurement procedures produce sufficiently comparable results for the intended purpose.

One commonly used approach is Bland-Altman analysis, which examines differences between paired measurements in relation to their average values.

This approach can help identify average differences, variability in differences, and patterns suggesting that agreement changes across the measurement range.

However, the analysis requires appropriate assumptions, and its interpretation depends on whether the observed differences are acceptable for the intended research or clinical purpose.

Correlation alone is not sufficient for establishing agreement. Two devices can correlate strongly while producing systematically different values.

In this hypothetical example, Device A reads an average of two units higher than Device B. Individual paired differences may nevertheless be substantially larger or smaller.

The limits do not establish whether the devices are interchangeable. That judgment requires predefined, substantively justified acceptance criteria and consideration of the assumptions and uncertainty surrounding the estimates.

Why Is Correlation Insufficient for Device Agreement?

Correlation measures the strength of association between variables, not whether their numerical values agree.

Suppose Device B always reports exactly ten units more than Device A. Their measurements could have a perfect positive correlation while exhibiting a systematic difference of ten units.

Likewise, two devices may rank participants similarly but produce differences large enough to affect classifications or substantive conclusions.

Researchers should distinguish agreement from association and select analyses that address the intended measurement question.

What Is Interlaboratory Comparison?

Interlaboratory comparison involves organizing, performing, and evaluating measurements or tests on the same or similar items by two or more laboratories under specified conditions.

Such comparisons can help identify systematic differences and evaluate laboratory performance.

Proficiency testing is one form of interlaboratory comparison used to evaluate participant performance against established criteria.

However, successful participation in a proficiency-testing program does not automatically establish equivalence for every analyte, measurement range, specimen type, or research application.

Researchers should examine whether the comparison materials and procedures are relevant to their study.

How Can Researchers Harmonize Measurements Across Sites?

Measurement harmonization aims to improve the comparability of results obtained through different procedures, instruments, or sites.

Possible approaches include common reference materials, standardized protocols, calibration procedures, centralized analysis, and planned cross-device comparisons.

In some studies, a subset of specimens or participants may be measured using more than one device or laboratory to estimate systematic differences.

These bridging measurements can support appropriate adjustment or sensitivity analysis when the measurement model and data are sufficient.

However, harmonization is not equivalent to forcing all measurements to have identical distributions.

Statistical adjustments that remove genuine population or site differences can introduce new bias. The harmonization strategy must distinguish measurement-related variation from substantive variation.

What Happens When Equipment Changes During a Longitudinal Study?

Equipment replacement, software updates, recalibration, and laboratory changes can create discontinuities in a measurement series.

For example, a researcher may observe an abrupt increase in recorded activity after a wearable device is replaced. The increase could reflect genuine behavioral change, a difference between devices, or both.

When possible, overlap measurements using the old and new systems can help evaluate comparability.

Researchers should also preserve records of equipment models, software versions, calibration dates, maintenance, and relevant procedural changes.

Without such information, it may be difficult to distinguish measurement discontinuities from genuine longitudinal trends.

What Quality-Control Procedures Should Be Established?

Quality control should be planned before data collection and tailored to the measurement system.

Relevant procedures may include calibration schedules, control materials, equipment maintenance, environmental monitoring, operator training, duplicate measurements, and documented responses to out-of-range results.

Laboratory quality-management frameworks, including ISO/IEC 17025 for testing and calibration laboratories and ISO 15189 for medical laboratories, provide relevant requirements within their respective scopes.

These standards should not be treated as universal requirements for every educational or social science research project. Their principles may nevertheless inform the design of proportionate measurement controls.

Researchers should specify the acceptance criteria and corrective actions that apply to their particular measurements.

Further planning of measurement quality-control procedures should address how errors will be detected and managed throughout data collection.

How Should Device and Laboratory Differences Be Reported?

Research reports should identify the measurement systems used and provide sufficient information for readers to evaluate their comparability.

Depending on the study, this may include device models, software versions, calibration procedures, laboratory methods, sample handling, measurement uncertainty, and quality-control results.

If multiple systems were used, explain how they were assigned to participants or sites and whether method-comparison or harmonization procedures were conducted.

Where systematic differences remain unresolved, describe their potential consequences for the intended comparisons.

Transparent reporting is especially important when equipment changes occur during a longitudinal study or when different laboratories process specimens from different research groups.

How Two Measurement Devices Can Create an Apparent Difference Between Research Sites

The example isolates a proportional measurement error. Actual wearable devices may differ in more complicated ways, including activity-specific accuracy, nonlinear response, and software processing. A simple percentage correction should not be applied without supporting evidence.

Common Misconceptions About Device and Laboratory Measurement Error

How to Manage Device and Laboratory Differences in Your Research

Begin by identifying every measurement system used in the study and determining whether device or laboratory assignment is associated with research groups, sites, or time points.

Then evaluate whether the systems produce measurements that are sufficiently comparable for the intended analysis.

Preserve calibration records, equipment identifiers, software versions, laboratory procedures, and relevant quality-control information.

In the methodology, describe the evidence supporting measurement comparability. In the discussion, acknowledge unresolved systematic differences that could affect the interpretation of results.

The objective is not to require identical equipment in every study. It is to ensure that differences between measurement systems do not become unrecognized explanations for the findings.

Before Combining Measurements From Different Devices or Laboratories

Frequently Asked Questions About Device and Laboratory Measurement Error

Measurement Systems Must Be Comparable Before Their Results Are Treated as Interchangeable

How to Cite This Guide

This guide is intended to be read, shared, and used in research, teaching, and academic work. If you draw on its ideas, explanations, or other content, please acknowledge the source by citing the guide. Doing so gives appropriate credit and helps your readers locate the original resource.