03 · What You Need to Know
How Duplicate Records and Identity-Linkage Errors Affect Research Evidence
What Is a Duplicate Record?
A duplicate record is an additional representation of an observation, entity, or event that should not be counted as a separate instance under the intended data model.
For example, a completed questionnaire may be imported twice into a research database. If both copies represent the same submission, treating them as independent responses would inflate the number of observations.
However, two identical questionnaire responses from different participants are not necessarily duplicates. Neither are repeated measurements from the same participant when each measurement represents a distinct occasion.
Researchers should therefore define what constitutes a unique participant, event, encounter, transaction, or measurement before searching for duplicates.
Distinguish Duplicate Records, Repeated Observations, and Merged Identities
Erroneous duplicate
Two or more records that represent the same underlying observation or event but have been stored or counted as separate instances.
Legitimate repeated observation
Multiple valid measurements or events associated with the same research unit at different occasions or under distinct conditions.
Merged identity
A record-linkage error in which information belonging to different individuals or entities is incorrectly combined under one identity.
Another important problem is a split identity, in which records belonging to one individual are incorrectly treated as belonging to different people.
These errors require different responses. Removing repeated observations may destroy valid longitudinal evidence, while retaining erroneous duplicates may inflate counts and distort statistical estimates.
Why Must Researchers Define the Unit of Analysis First?
The unit of analysis determines what counts as one observation in the study.
A dataset may contain one row per student, one row per course enrollment, one row per assessment, or one row per learning platform event.
These structures have different expectations concerning repeated identifiers.
For example, a student appearing in six academic-term records may be entirely appropriate in a longitudinal dataset. The same student appearing twice in a table intended to contain one record per individual may require investigation.
Researchers should document the expected relationship between identifiers and observations before applying deduplication rules.
How Do Duplicate Records Enter Digital Systems?
Duplicate records may arise through repeated submissions, interrupted synchronization, data migration, overlapping exports, manual entry, or inconsistent identifier assignment.
For example, an offline survey application may retransmit a record after an interrupted upload. If the receiving system creates a new record rather than recognizing the original submission, the dataset may contain two copies.
Similarly, combining annual administrative exports may duplicate records included in overlapping reporting periods.
Understanding the source of duplication helps determine whether the problem is isolated or affects an identifiable set of records.
What Types of Identity Errors Should Researchers Look For?
| Problem |
Illustrative situation |
Potential consequence |
| Exact duplicate |
One survey submission imported twice |
Inflated response count |
| Near duplicate |
Same event stored with different formatting |
Repeated event counted separately |
| Split identity |
One student assigned two identifiers |
Fragmented longitudinal history |
| False merge |
Two patients linked under one identifier |
Mixed clinical histories |
| Incorrect event linkage |
Assessment assigned to the wrong participant |
Outcome misclassification |
| Repeated valid event |
Two separate visits on the same day |
Loss of evidence if incorrectly deleted |
| Overlapping extracts |
Monthly exports include previously extracted transactions |
Inflated event totals |
These categories illustrate why a single rule such as removing identical rows is insufficient for many research datasets.
Why Are Matching Names Not Enough to Establish Identity?
Names may be shared by different individuals, entered inconsistently, abbreviated, misspelled, or changed over time.
Dates of birth and other demographic characteristics may also be shared or incorrectly recorded.
For example, two students with the same name may be different individuals, while one student may appear under different versions of their name.
Researchers should avoid merging records solely because names or a few demographic fields match.
Identity resolution should use identifiers and linkage evidence appropriate to the data source and intended purpose.
What Is Record Linkage?
Record linkage is the process of determining which records from one or more datasets refer to the same underlying person, organization, event, or entity.
It may rely on unique identifiers, combinations of attributes, or statistical matching procedures.
Deterministic linkage uses specified matching rules, such as agreement on a reliable unique identifier.
Probabilistic linkage evaluates evidence from several fields and estimates how strongly their agreement or disagreement supports a match.
Neither approach guarantees correctness. Their suitability depends on identifier quality, data completeness, and the characteristics of the records.
How Do False Matches and Missed Matches Differ?
A false match occurs when records belonging to different entities are incorrectly linked.
A missed match occurs when records belonging to the same entity are not linked.
Both can distort research results, but their consequences differ.
A false match may combine different individuals' exposures and outcomes, creating a misleading longitudinal history.
A missed match may fragment one individual's observations across separate identities, potentially reducing observed follow-up or inflating participant counts.
Researchers should evaluate both types of error rather than optimize linkage solely to maximize the number of matches.
Can Researchers Use Exact Matching to Remove Duplicates?
Exact matching can be useful when a stable, reliable identifier uniquely represents the intended event or observation.
For example, two records sharing the same immutable submission identifier may represent the same questionnaire submission, provided the system guarantees that the identifier is unique and the records have compatible provenance.
However, exact agreement on all visible fields does not necessarily establish duplication.
Two participants may legitimately provide identical answers, and two distinct transactions may have identical amounts and dates.
Conversely, erroneous duplicates may differ in formatting, timestamps, or subsequently edited fields.
Exact matching should therefore be interpreted according to the data model and source-system rules.
When Is Probabilistic Record Linkage Appropriate?
Probabilistic linkage may be useful when no reliable common identifier exists and several imperfect attributes are available.
The classical Fellegi-Sunter framework evaluates agreement patterns to support decisions about whether record pairs refer to the same entity.
For example, matching may consider names, dates of birth, institutional identifiers, or other authorized fields.
However, the use of identifying information must be justified and appropriately protected.
Linkage thresholds should reflect the consequences of false and missed matches, and uncertain pairs may require authorized review.
Researchers should not assume that a high algorithmic similarity score establishes identity beyond reasonable uncertainty.
How Can Merged Identities Distort Longitudinal Research?
Longitudinal research depends on linking repeated observations to the correct research unit.
If two individuals' records are incorrectly merged, the combined history may contain artificial changes or impossible sequences.
For example, a merged student record might show simultaneous enrollment in incompatible programs or conflicting academic statuses.
However, unusual patterns are not definitive proof of an identity error. Legitimate transfers, dual enrollment, and administrative corrections may produce complex histories.
Researchers should investigate implausible sequences using source documentation and authorized data stewards before altering identities.
How Do Duplicate Records Affect Statistical Analysis?
Erroneous duplication can distort counts, proportions, estimated associations, and measures of uncertainty.
For example, duplicating observations from participants with particular outcomes may change estimated prevalence or group differences.
Even when duplication does not change a sample mean, treating repeated copies as independent observations may produce artificially narrow uncertainty estimates.
Identity errors can also affect clustering, longitudinal models, survival analyses, and the calculation of participant-level outcomes.
The direction and magnitude of bias depend on which records are affected and how they enter the analysis.
Researchers should not assume that duplication always inflates the substantive estimate or that its consequences are negligible when the proportion of affected records is small.
Should Researchers Delete Duplicate Records Immediately?
Generally, suspected duplicates should first be flagged and investigated rather than permanently deleted.
A defensible workflow preserves the original authorized dataset, records the criteria used to identify possible duplicates, and documents subsequent decisions.
Confirmed erroneous copies may be excluded from an analytical dataset while retaining an appropriate audit trail.
When records contain complementary information about the same event, a carefully defined reconciliation procedure may be necessary instead of simply retaining the first or last entry.
Unresolved cases should not be silently removed.
What If Two Records Contain Conflicting Information?
Conflicting values require investigation of their provenance and meaning.
For example, two records sharing a submission identifier may contain different assessment scores because one represents a later authorized correction.
The correct analytical value may depend on the study's definition of the outcome and the source system's correction rules.
Automatically retaining the newest record may be inappropriate if the later timestamp reflects migration rather than a genuine correction.
Similarly, averaging conflicting values may produce a number that was never observed.
Researchers should establish explicit precedence and reconciliation rules supported by the source documentation.
How Can Duplicate Detection Be Validated?
Researchers may evaluate a justified sample of records classified as matches, nonmatches, or uncertain pairs against appropriate reference information.
This can help estimate false-match and missed-match errors.
Validation should consider relevant groups, data sources, and periods because linkage performance may differ across them.
For example, matching based on names may perform differently when names are incomplete or recorded inconsistently.
Reviewers should use standardized criteria and appropriate privacy safeguards.
The validation sample should be selected to support the intended quality assessment rather than merely confirm obvious matches.
How Should Researchers Handle Uncertain Matches?
Some record pairs cannot be classified confidently using the available information.
Researchers may retain an uncertain status, seek additional authorized evidence, or evaluate the consequences of alternative linkage decisions.
For example, sensitivity analyses might compare results under conservative and more inclusive linkage rules.
However, these analyses do not establish which identities are correct. They indicate how strongly the substantive conclusions depend on unresolved linkage assumptions.
When uncertainty affects critical outcomes or participant histories, additional validation may be necessary.
What Role Does Data Provenance Play?
Data provenance records where information originated and how it was processed.
For duplicate and identity-resolution work, useful provenance may include source-system identifiers, extraction dates, record creation times, migration histories, and transformation rules.
Maintaining this information helps distinguish repeated extracts from repeated events.
It also supports reproducibility when researchers need to reconstruct why records were linked, separated, or excluded.
Original identifiers and source records should be protected according to applicable privacy and data-use requirements.
How Can Privacy Be Protected During Identity Resolution?
Record linkage may require identifiable information that is unnecessary for the final analytical dataset.
Researchers should minimize access to such information and use authorized, secure linkage procedures.
Where feasible, an approved data steward or trusted linkage service may perform identity resolution and provide appropriately coded research identifiers.
Hashing names or identifiers does not automatically guarantee anonymity, particularly when the underlying values are predictable or linkable.
The appropriate safeguards depend on the data, institutional arrangements, applicable law, and approved research protocol.
Should Duplicate Records Be Reported in the Research Article?
Researchers should report duplicate identification and resolution procedures when they materially affect the study population, data quality, or analysis.
Relevant details may include the number of records examined, the criteria used to identify duplicates, the number excluded or reconciled, and unresolved linkage uncertainty.
For studies using routinely collected health data, the RECORD statement provides relevant reporting guidance concerning data sources, linkage, and participant selection.
Researchers should also explain whether identity resolution changed the number of unique participants or observation periods.
Transparent reporting allows readers to evaluate the consequences of the data preparation process.
When Can Identity Problems Make a Dataset Unsuitable?
A dataset may be unsuitable when critical identities cannot be distinguished, linkage errors are extensive, or the available evidence cannot support reliable participant-level histories.
For example, a study estimating individual academic progression may be compromised if records from different students have been merged and cannot be separated.
A study using only aggregate event counts may be less affected by some identity errors, although duplication can still distort totals.
The decision depends on the intended unit of analysis and the consequences of unresolved errors.
Watch Out
Do not remove every repeated identifier or merge records solely because their names and demographic characteristics match. Legitimate repeated observations, erroneous duplicates, split identities, and false merges require different treatments. Incorrect deduplication can create new errors while appearing to clean the dataset.
