03 · What You Need to Know
How to Evaluate Whether Existing Data Are Fit for Prognostic Research
What Counts as Existing Data for Prognostic Research?
Existing data are records collected before the current research analysis, often for clinical care, administration, surveillance, or an earlier research project.
Examples include electronic health records, disease registries, insurance claims, administrative databases, established longitudinal cohorts, and institutional student records.
These sources differ in their original purpose, measurement procedures, population coverage, and outcome ascertainment.
A clinical registry may contain detailed disease characteristics but incomplete information about events occurring elsewhere. Administrative records may provide broad coverage while lacking clinically important measurements.
Researchers should evaluate the strengths and limitations of the particular source rather than treating all secondary datasets as methodologically equivalent.
Can Retrospective Data Support a Prognostic Study?
Yes. Prognostic research does not necessarily require researchers to begin collecting data before future outcomes occur.
A retrospective cohort may reconstruct an appropriate historical starting point and examine outcomes recorded afterward.
For example, researchers could identify patients diagnosed between 2018 and 2022, extract characteristics measured at diagnosis, and determine subsequent outcomes through available records.
The study remains longitudinal in its analytical structure even though the information already exists when the research begins.
However, retrospective availability does not guarantee that the records establish the required temporal sequence accurately.
The adequacy of the design depends on how well the historical cohort and its follow-up can be reconstructed.
What Must an Existing Dataset Contain?
The essential requirements follow from the prognostic question.
| Data Requirement |
Why It Matters |
Potential Problem |
| Defined source population |
Establishes who the findings describe |
Records include only selected or surviving individuals |
| Reliable time zero |
Establishes the start of outcome risk |
Diagnosis and registration dates are confused |
| Appropriately timed predictors |
Ensures information was available at the intended prediction point |
Post-outcome or post-treatment measurements are treated as baseline |
| Valid outcome definition |
Identifies the event being estimated |
Administrative codes do not reliably represent the target outcome |
| Outcome dates or status |
Supports the specified horizon and analysis |
Events are incompletely recorded or incorrectly dated |
| Adequate follow-up |
Provides information about subsequent outcomes |
Most participants have insufficient observation time |
| Relevant covariates |
Supports factor analysis or prediction |
Important measurements are unavailable or inconsistent |
| Data provenance |
Supports reproducibility and interpretation |
Variable definitions or coding changes are undocumented |
A dataset does not need every conceivable variable. It needs sufficiently reliable information for the specific objective and analytical strategy.
Why Is the Source Population Important?
Existing databases rarely contain a random sample of everyone who could experience the outcome.
Hospital records include people who received care at that institution. Specialist registries may disproportionately represent severe or complex conditions. Student information systems may exclude individuals who never completed enrollment.
These selection mechanisms affect which people are represented and potentially the relationships observed among predictors and outcomes.
For example, a registry of patients who survived long enough to attend follow-up cannot automatically estimate prognosis from diagnosis for all newly diagnosed patients.
Researchers should identify the process through which individuals enter the database and determine whether that process corresponds to the intended prognostic population.
Why Is Time Zero Especially Important in Existing Records?
Routine datasets may contain several dates that appear to describe the beginning of a condition.
These can include the first recorded diagnosis code, first consultation, hospital admission, registry entry, or treatment initiation.
However, the first date recorded in a database may occur after the actual clinical event.
For example, a patient diagnosed elsewhere may enter a specialist registry months later. Treating registry entry as the diagnosis date would change the prognostic starting population.
Researchers should determine whether the data support a defensible definition of time zero and whether delayed entry or survivor selection requires attention.
When the starting event cannot be established reliably, some prognostic questions may not be answerable from the available records.
How Can Existing Data Create Information Leakage?
Information leakage occurs when predictors include information that would not have been available at the intended prediction point.
Retrospective datasets make this problem particularly easy to overlook because measurements from different dates may appear together in a single analytical table.
For example, a dataset might contain the highest laboratory value recorded during hospitalization. If the model is intended for use at admission, that value may incorporate information obtained after admission.
Similarly, a student's final-semester attendance cannot legitimately be used to predict withdrawal risk at initial enrollment.
Researchers should reconstruct predictor availability using actual timestamps and the intended decision point.
Variables that summarize future events should not be treated as baseline predictors merely because they are stored in the same record.
How Should Predictor Measurement Quality Be Evaluated?
Researchers should examine how each predictor was defined, collected, coded, and stored.
Routine measurements may vary across clinicians, departments, institutions, or calendar periods.
For example, a laboratory assay may change during the study period. A clinical severity score may be recorded only for patients considered seriously ill. An educational engagement measure may be calculated differently after a learning management system update.
Such differences can introduce measurement error or systematic differences between participant groups.
Researchers should review data dictionaries, source documentation, measurement protocols, and relevant changes in operational practice.
When necessary, they should assess whether harmonization is scientifically defensible.
What If a Predictor Was Measured Only in Selected Participants?
Selective measurement can create missing-data and selection problems.
For example, a laboratory test may be ordered primarily for patients with severe symptoms. Its absence among other patients may therefore reflect clinical judgment rather than random omission.
Restricting the analysis to patients with available results may change the study population.
Alternatively, the fact that a test was ordered may itself contain information about the clinical process.
Researchers should investigate why measurements are missing and whether the measurement process would be reproduced in the intended application.
A predictor that is rarely available in routine practice may also limit the practical usefulness of a prediction model.
How Can Outcome Ascertainment Be Incomplete?
Existing databases may fail to capture outcomes occurring outside the system that generated the records.
For example, a hospital database may record readmissions to the same hospital but not admissions elsewhere.
A university database may identify withdrawal from one institution without recording transfer to another institution.
Such omissions can lead researchers to misclassify outcomes or interpret an absence of recorded events as evidence that no event occurred.
The problem becomes particularly serious when the probability of missing an outcome differs across participant groups.
Researchers should determine the actual coverage of the data source and, where feasible, consider linkage to complementary registries or records.
Is an Administrative Code a Valid Outcome Definition?
Not automatically. Administrative codes are often created for billing, service management, or reporting rather than research classification.
A code may have imperfect sensitivity or specificity for the clinical event researchers want to study.
Definitions may also change over time or differ across institutions.
Researchers should seek validation evidence for coding algorithms when available and determine whether those findings apply to their dataset.
Where feasible, chart review or another appropriate reference assessment may help evaluate classification accuracy.
The outcome definition should reflect the research question rather than being selected solely because a convenient database field exists.
How Much Follow-Up Must Existing Data Provide?
The available observation period must support the intended prognostic horizon.
A dataset spanning ten calendar years does not necessarily provide ten years of follow-up for every participant.
Individuals entering near the end of the observation period may contribute only a few months.
Researchers should examine the distribution of follow-up times, event dates, censoring, and numbers remaining at risk at relevant horizons.
For fixed-time outcomes, the completeness of outcome ascertainment at the target horizon is especially important.
Methods for handling censoring cannot automatically compensate for insufficient information about late outcomes.
The principles of choosing follow-up duration in prognostic studies apply equally to existing datasets.
What Is Informative Censoring in Routine Data?
Censoring may be informative when the probability that follow-up ends is related to future outcome risk after accounting for the variables included in the analysis.
For example, patients with worsening symptoms may transfer to another hospital, causing their subsequent outcomes to disappear from the original institution's records.
Similarly, students experiencing academic difficulties may stop using institutional systems before formal withdrawal is recorded.
Standard survival methods rely on assumptions concerning censoring that may not hold in these circumstances.
Researchers should investigate the mechanisms underlying incomplete observation and consider appropriate analytical or sensitivity approaches.
The absence of an event code should not automatically be interpreted as event-free survival.
How Should Missing Predictor Data Be Handled?
Missing predictor values require investigation before selecting an analytical method.
Researchers should describe the extent and patterns of missingness and consider why values are unavailable.
Complete-case analysis may be inefficient or biased when participants with complete records differ systematically from those with missing information.
Multiple imputation may be appropriate under suitable assumptions and with an appropriately specified imputation model.
However, imputation does not automatically recover information missing because of unmeasured clinical decisions or systematic data collection failures.
For prediction-model research, the handling of missing predictors during development should also be compatible with how missing information will be managed when the model is applied.
Can Large Datasets Overcome Poor Data Quality?
No. Increasing sample size may reduce random sampling uncertainty, but it does not automatically remove systematic measurement error, selection bias, or outcome misclassification.
For example, a database containing 100,000 records may estimate the wrong outcome probability very precisely if events occurring outside the institution are consistently missed.
Similarly, a prediction model trained on thousands of observations may perform poorly in another setting if its predictors encode local documentation practices.
Large datasets can be valuable, but their size should not be confused with validity.
Researchers should evaluate whether systematic limitations are likely to affect the intended estimates and conclusions.
How Do Changes Over Calendar Time Affect Prognostic Research?
Existing datasets often span periods during which clinical practice, institutional policies, measurement systems, or population characteristics change.
For example, new treatments may reduce mortality, while revised diagnostic criteria may change who enters a disease registry.
In educational settings, changes in curriculum, assessment policy, or student-support systems may alter the relationship between baseline characteristics and subsequent outcomes.
Researchers should identify important changes and consider whether data from different periods remain comparable.
Calendar-time effects may be relevant to both prognostic factor interpretation and model transportability.
What If Data Come From Several Institutions?
Multicenter datasets can improve coverage and provide opportunities to examine differences across settings.
However, institutions may use different coding systems, measurement procedures, eligibility criteria, and outcome ascertainment practices.
Combining records without addressing these differences can introduce heterogeneity that is difficult to interpret.
Researchers should harmonize definitions where justified and examine whether important differences remain.
For prediction models, internal-external validation approaches may help evaluate performance across participating clusters or settings.
Multiple institutions do not automatically guarantee representativeness or external validity.
When Is Existing Data Adequate for Prognostic Factor Research?
A dataset may support prognostic factor research when the factor of interest is measured appropriately, the outcome occurs afterward, and the relevant population and covariates can be defined.
Researchers should consider whether important confounding or selection mechanisms can be addressed, especially when explanatory or causal interpretations are intended.
For example, an association between baseline severity and later recovery may be estimable from routine records if severity is recorded consistently and recovery is adequately ascertained.
However, a missing measure of an important determinant may limit the interpretation of the adjusted association.
The required information depends on the specific estimand rather than a generic checklist of desirable variables.
When Is Existing Data Adequate for Prediction-Model Development?
Prediction-model development requires information beyond a suitable outcome and several available predictors.
Researchers need sufficient outcome information, appropriate candidate predictors, a defined prediction time, and adequate data for estimating and validating the model.
Predictors should be available at the intended application point and measured in a way that can be reproduced.
The number of candidate predictor parameters should be supported by the available sample size and outcome events.
Researchers should also plan appropriate internal validation and evaluate whether the model can be externally validated.
The requirements for outcome events and prognostic sample size should be considered before fitting a complex model.
When Is Existing Data Better Used for External Validation?
An existing dataset may be particularly valuable for evaluating a previously developed model.
If the data contain the necessary predictors, an appropriate outcome, and sufficient follow-up, researchers may apply the fixed model and assess performance.
This can provide evidence about whether predictions remain accurate in another population or period.
However, validation requires the original model to be implemented correctly and without outcome-driven modification before its initial performance is assessed.
Researchers should determine whether model development, validation, or updating offers the most useful contribution before beginning the analysis.
How Should Data Adequacy Be Evaluated Before Analysis?
A structured feasibility assessment can identify limitations before researchers commit to a particular design.
Researchers should first translate the research question into explicit eligibility, time-zero, predictor, outcome, and follow-up definitions.
They should then examine data dictionaries, coding practices, missingness, event frequencies, and observation periods.
Where feasible, preliminary checks should assess the reliability of key dates and outcome classifications.
Such feasibility work should be distinguished from outcome-driven model optimization or selective hypothesis testing.
If the dataset cannot support the intended question, researchers should consider modifying the research objective, obtaining complementary data, or collecting new information.
What Ethical and Governance Requirements Apply?
Using existing records does not automatically eliminate ethical or data-protection obligations.
Requirements depend on the jurisdiction, institution, data sensitivity, consent arrangements, and applicable research governance framework.
Researchers may need ethics review, data access authorization, data-sharing agreements, security controls, or other approvals.
De-identification can reduce certain privacy risks but does not necessarily remove all governance requirements.
Researchers should verify applicable institutional and legal requirements before accessing or linking records.
Methodological feasibility and permission to use the data are separate questions, and both must be addressed.
How Should Research Using Existing Data Be Reported?
Researchers should describe the data source, its original purpose, population coverage, observation period, and procedures for identifying eligible participants.
They should report how time zero, predictors, outcomes, and follow-up were reconstructed.
Missingness, coding limitations, linkage procedures, and important changes in data collection should be documented.
STROBE provides general reporting guidance for observational studies. RECORD extends reporting considerations for studies using routinely collected health data.
For prediction-model studies, applicable TRIPOD+AI guidance should also be considered.
Transparent reporting allows readers to evaluate whether the available records genuinely support the prognostic conclusions.
