03 · What You Need to Know
How to Verify Research Data Management Before the Main Study
What Does Pilot Testing Data Management Involve?
Data management pilot testing involves rehearsing the procedures through which research information is recorded, checked, organized, protected, transferred, and prepared for analysis.
It may begin with a participant's response or an observation and continue through database entry, validation, storage, cleaning, and export.
For example, a researcher may enter synthetic survey responses, export the records, and verify that the values match the original entries.
Another study may test whether interview identifiers can be linked securely to survey records without exposing participants' identities to unauthorized personnel.
The purpose is to identify errors and weaknesses before they affect a substantial volume of research data.
Why Is Data Management Piloting Important?
Research findings depend on the integrity of the information used for analysis.
Even well-designed instruments can produce unusable datasets when responses are recorded incorrectly, variables are mislabeled, identifiers are inconsistent, or essential information is lost.
For example, a survey may record a five-category response scale correctly on screen but export the categories in an order that the researcher interprets incorrectly.
If this problem remains undetected, subsequent analysis may produce misleading scores.
Similarly, missing participant identifiers may prevent longitudinal observations from being connected.
Data management testing helps determine whether the planned workflow preserves the meaning, completeness, and traceability of the original evidence.
Which Data Management Procedures Should Be Tested?
| Procedure |
What to verify |
Potential problem |
| Data entry |
Whether recorded values match source information |
Transcription or entry errors |
| Variable coding |
Whether values, labels, and missing-data codes are defined correctly |
Misinterpretation of categories |
| Validation |
Whether invalid, inconsistent, or duplicate entries are detected |
Undetected data quality problems |
| Participant identifiers |
Whether identifiers are unique and stable where required |
Incorrect linkage or duplicate records |
| Longitudinal linkage |
Whether observations from different time points can be matched |
Lost or incorrectly matched follow-up data |
| File storage |
Whether records are stored in approved locations with appropriate permissions |
Unauthorized access or accidental loss |
| Backup and recovery |
Whether recoverable copies exist and restoration works |
Irrecoverable data loss |
| Data export |
Whether exported records preserve values, labels, and formats |
Corrupted or incorrectly interpreted files |
| Analysis preparation |
Whether the resulting dataset supports the planned procedures |
Missing variables or incompatible formats |
The scope should reflect the actual study. A simple anonymous questionnaire may require a narrower verification process than a multisite longitudinal mixed-methods investigation.
What Is the Difference Between Data Entry Testing and Data Management Testing?
Data entry testing focuses on whether information is captured accurately.
Data management testing examines the broader lifecycle of that information, including storage, coding, access, linkage, cleaning, transfer, and preparation for analysis.
Data entry testing
Verifies that information entered or captured through an instrument is recorded accurately and consistently.
Data management testing
Evaluates the procedures used to organize, validate, protect, preserve, transform, and prepare research data throughout the study.
Both are important, but successful entry does not guarantee that later processing will preserve the data correctly.
When Is Data Entry Piloting Especially Necessary?
Data entry piloting is particularly valuable when information must be transferred manually from paper instruments, observation forms, or other sources into a database.
Manual entry can introduce transcription errors, omitted values, and inconsistent coding.
Researchers may test whether entry forms contain suitable validation rules and whether staff understand the codebook.
For example, a questionnaire response of "Strongly disagree" should be mapped to the intended numerical value consistently across all records.
However, electronic capture does not eliminate data quality risks. Online forms may contain faulty validation rules, incorrectly configured skip logic, or export problems.
Should You Use Double Data Entry?
Independent double data entry may be useful when manual transcription errors present a consequential risk.
Two individuals enter the same source information independently, and discrepancies are compared and resolved against the original record.
However, double entry is not universally necessary.
For electronic instruments that capture responses directly, other verification procedures may be more efficient.
The decision should reflect the risk of error, importance of the variables, available resources, and reliability of the data capture process.
Double entry also does not eliminate errors already present in the original source.
How Should Variable Coding Be Tested?
Researchers should establish a data dictionary or codebook defining variable names, descriptions, permissible values, missing-data codes, and relevant derivations.
For example, a five-point agreement scale may use values from 1 to 5, but the meaning of each value must remain consistent across data entry, export, and analysis.
A pilot can verify that the coding system is implemented correctly.
It may also identify ambiguous categories or inconsistent naming conventions.
Researchers should not assume that the same numerical value has the same meaning across different variables.
The codebook should describe the intended interpretation rather than rely on undocumented conventions.
Should Missing-Data Codes Be Tested?
Yes. Missingness can arise because participants skip questions, are ineligible for particular items, withdraw, or encounter technical problems.
These situations may have different meanings.
For example, a question intentionally skipped because it does not apply to a participant should not automatically be treated as equivalent to an unanswered applicable question.
Researchers should determine how each type of missingness will be represented and whether the database preserves the distinction.
A pilot can test whether missing values are exported correctly and whether the analysis software interprets them as intended.
Using ordinary numerical values as missing-data codes without proper handling can distort descriptive statistics and model estimates.
How Can Validation Rules Be Tested?
Validation rules identify values that fall outside expected ranges or violate specified conditions.
For example, a form may require a participant age within an approved eligibility range or prevent duplicate participant identifiers.
Researchers can test whether the rules detect invalid entries and whether they incorrectly reject legitimate values.
However, overly restrictive validation can also create problems.
A researcher should not automatically prevent unusual but plausible observations merely because they differ from expectations.
Validation should distinguish impossible values, inconsistent records, and potentially valid outliers requiring review.
Should You Test Participant Identifiers?
Yes, when the study requires repeated observations or linkage across data sources.
Identifiers should be sufficiently unique and stable to support the intended connections.
For example, a longitudinal study may need to link baseline and follow-up questionnaires without including participant names in the analysis dataset.
A pilot can examine whether identifiers are generated consistently, whether duplicates are detected, and whether records can be matched correctly.
Researchers should also establish how the linkage key is protected and who is authorized to access it.
Participant identifiers are not merely administrative conveniences. Incorrect linkage can create substantive analytic errors and privacy risks.
How Should Longitudinal Data Management Be Piloted?
Longitudinal studies collect information at multiple time points and require consistent identification of participants, assessments, and dates.
A pilot may test whether follow-up records can be matched to baseline data, whether missed assessments are recorded appropriately, and whether repeated measures are stored in the intended structure.
For example, a semester-long study may collect engagement measures at baseline, midterm, and the end of the term.
The database must distinguish these time points accurately.
Researchers should also consider changes in participant status, withdrawal, and incomplete follow-up.
Errors in longitudinal linkage can undermine analyses even when individual questionnaire responses were recorded correctly.
How Should Data Linkage Be Tested in Mixed-Methods Research?
Mixed-methods studies may need to connect quantitative records with qualitative interviews or observations.
For example, an explanatory sequential study may select interview participants based on their survey profiles.
The data management system must support the intended connection without exposing unnecessary identifying information.
A pilot can test whether participant identifiers remain consistent, whether recontact permissions are recorded, and whether qualitative materials can be associated with the correct quantitative cases.
However, linkage should be consistent with informed consent and applicable ethics requirements.
Researchers should not describe a survey as irreversibly anonymous if a mechanism exists to reconnect records to identifiable individuals.
Can Synthetic Data Be Used for Testing?
Yes. Synthetic data are artificially constructed records used to test procedures without relying on actual participant information.
They can be particularly useful for examining database structures, validation rules, import and export procedures, identifier formats, and analysis scripts.
For example, researchers can create records containing valid values, missing values, duplicate identifiers, and intentionally invalid entries to examine whether the system handles them correctly.
However, synthetic data do not fully reproduce real participant behavior or every irregularity found in actual data.
They should be designed to test relevant edge cases rather than only ideal records.
Realistic pilot data may still be necessary to evaluate some operational processes.
What Are Edge Cases in Data Management Testing?
Edge cases are unusual but plausible conditions that may expose weaknesses in a workflow.
Examples include participants with missing follow-up records, duplicate submissions, unexpected text characters, unusual date formats, or responses that legitimately fall near the boundaries of permitted ranges.
For example, a data import process may work correctly for ordinary English text but fail when participant responses contain accented characters or non-Latin scripts.
Another workflow may incorrectly interpret a date recorded in day-month-year format as month-day-year.
Testing such conditions can identify errors that routine examples may not reveal.
Should Data Export Be Piloted?
Yes, particularly when data move between collection platforms and analysis software.
Researchers should verify that exported values, labels, identifiers, dates, and missing-data codes retain their intended meaning.
For example, a survey platform may export response labels as text while the analysis script expects numerical codes.
Another platform may change variable names or represent unanswered questions differently across export formats.
A pilot should include opening the exported file in the intended analysis environment and checking its contents against known source records.
Successful file creation does not guarantee that the exported dataset is analytically correct.
Should You Test the Analysis Pipeline at the Same Time?
Often, yes, when the analysis depends on specific data structures or transformations.
For example, a researcher may need to reverse-code questionnaire items, calculate scale scores, merge datasets, or generate longitudinal summaries.
Testing these procedures with synthetic or suitable pilot data can reveal programming errors and missing information requirements.
However, technical execution does not establish that the statistical analysis is methodologically appropriate.
The pipeline may run without errors while using an unsuitable model or incorrectly defined construct.
The related issue is examined in Should You Test Your Data Analysis Pipeline Using Dummy or Pilot Data?.
How Should Qualitative Data Management Be Tested?
Qualitative data management may involve audio recordings, transcripts, field notes, pseudonyms, analytic memos, and document versions.
A pilot can examine whether recordings are captured clearly, transferred securely, transcribed accurately enough for the intended analysis, and organized consistently.
Researchers should establish naming conventions and procedures for separating identifying information from analytic materials where appropriate.
However, qualitative data quality should not be reduced to standardized file completeness alone.
Context, reflexive documentation, and preservation of relevant meaning may be essential to the chosen methodology.
Should Automated Transcription Be Tested?
Yes, when automated transcription will be used.
Researchers should assess whether the service handles relevant accents, languages, technical terminology, overlapping speech, and recording conditions adequately.
Automated transcripts may contain omissions or substitutions that affect interpretation.
For example, a transcription error involving a negation can reverse the apparent meaning of a participant's statement.
Researchers should establish appropriate verification procedures and consider whether the service's data processing arrangements meet confidentiality and institutional requirements.
Automated transcription output should not automatically be treated as an accurate representation of the original recording.
How Should Data Security Be Tested?
Data security testing should examine whether the planned workflow protects information according to the study's risk profile and applicable requirements.
Relevant procedures may include access permissions, secure transfer, approved storage locations, backup arrangements, and separation of identifying information.
Researchers should verify that authorized personnel can access necessary information while unauthorized access is appropriately restricted.
However, pilot testing does not replace institutional security review or legal compliance.
Where sensitive personal information is involved, researchers should use approved systems and consult relevant institutional specialists.
Should Backup and Recovery Procedures Be Tested?
Yes, when data loss would materially affect the study.
Having a backup policy is not equivalent to demonstrating that records can be recovered.
Researchers may test restoration using synthetic or otherwise appropriate non-sensitive data.
For example, a team could verify whether a deleted test dataset can be restored with its variable definitions and associated files intact.
Recovery testing should not involve deliberately risking actual participant records.
Backup frequency, retention, and access arrangements should reflect institutional requirements and the importance of the information.
What Is the Role of a Data Management Plan?
A data management plan documents how research data will be handled throughout the study and, where applicable, preserved or shared afterward.
It may address collection, organization, documentation, storage, security, access, retention, and disposal.
Pilot testing evaluates whether selected procedures described in the plan work in practice.
For example, the plan may specify that survey and interview records will be linked using pseudonymous identifiers. The pilot can verify whether that linkage is accurate and secure.
A written plan is useful, but its existence does not establish operational feasibility.
How Do FAIR Principles Relate to Data Management?
The FAIR principles concern making data findable, accessible, interoperable, and reusable under appropriate conditions.
They encourage attention to documentation, metadata, identifiers, and technical compatibility.
However, FAIR does not mean that all research data should be publicly accessible.
Access may be restricted because of confidentiality, consent, legal obligations, or other legitimate considerations.
Pilot testing can examine whether documentation and file structures support the intended future use while respecting those restrictions.
Wilkinson and colleagues provide the foundational formulation of the FAIR Guiding Principles.
Should You Pilot Data Management in Multisite Studies?
Multisite studies may require particular attention because different locations can use inconsistent procedures, software versions, or coding conventions.
For example, one institution may record dates differently from another, or sites may apply different labels to the same questionnaire response.
A pilot can examine whether records from multiple sites can be combined without losing meaning.
Researchers should establish shared data definitions and procedures for resolving discrepancies.
However, central standardization should not erase legitimate contextual differences that are relevant to the research.
How Can Data Quality Be Assessed During the Pilot?
Data quality may be evaluated through accuracy, completeness, consistency, uniqueness, and suitability for the intended analysis.
Researchers should define indicators relevant to the actual workflow.
For example, they may calculate the proportion of records missing a required variable or count duplicate participant identifiers.
However, a dataset can be complete and internally consistent while still failing to represent the intended construct adequately.
Data management quality and measurement validity should therefore be evaluated separately.
Should You Establish Data Quality Thresholds?
Thresholds may be useful when they are linked to consequential feasibility decisions.
For example, a study may require a sufficiently high proportion of complete primary outcome records to support the definitive analysis.
However, thresholds should reflect the research design, analytic requirements, and practical consequences of missing information.
Researchers should avoid adopting arbitrary percentages as universal standards.
Some errors, such as incorrect participant linkage or unauthorized disclosure, may be unacceptable regardless of their frequency.
Progression decisions should consider the nature and consequences of problems, not only aggregate error rates.
What If the Pilot Reveals Data Management Errors?
Researchers should identify the source of the error and determine whether it affects data already collected.
For example, an incorrect export mapping may require correction of the codebook, revision of the export procedure, and verification of existing records.
After substantial changes, the workflow should be retested.
Researchers should document the problem, corrective action, and evidence that the revised procedure works.
Errors should not be concealed merely because they were discovered during preliminary testing.
Can Pilot Data Be Used in the Main Study?
Sometimes, depending on the design, consent arrangements, and compatibility of the data collection and management procedures.
However, if the pilot reveals that records were coded incorrectly or collected using substantially different procedures, combining them with definitive data may be inappropriate.
Researchers should distinguish recoverable technical errors from differences that change the meaning of the observations.
The decision should follow the study's methodological and ethical requirements rather than the desire to increase sample size.
How Should Data Management Pilot Findings Be Reported?
The report should describe the workflows tested, the types of records used, the errors identified, and the modifications implemented.
Where quantitative quality indicators are relevant, researchers should report their definitions and observed values.
They should also distinguish technical verification from evidence about real-world operational feasibility.
For example, successful testing with synthetic records may establish that an import script handles specified conditions, but it does not prove that all future participant data will be error-free.
The report should explain how the findings improve readiness for the definitive study.
Watch Out
Do not assume that a database is ready merely because records can be entered and exported. A technically successful workflow may still contain incorrect coding, missing linkage information, inadequate access controls, or transformations that alter the meaning of the original data.
