When Should a Prognostic Model Be Developed, Validated, or Updated?

Do You Really Need to Develop Another Prognostic Model?

You have access to a longitudinal dataset containing baseline characteristics and subsequent outcomes. You could use regression or machine learning to develop a model that estimates the probability of a future event. The dataset contains several promising predictors, and a new model seems like a reasonable research contribution.

But what if several prognostic models already exist for the same outcome? Should you build another one, evaluate an existing model in your population, or modify a model that performs poorly?

These choices have different methodological requirements and scientific implications. A newly developed model may appear impressive in its original dataset while providing little improvement over an existing model that has never been adequately evaluated outside its development setting.

Before choosing an analytical method, researchers need to determine what kind of prediction-model evidence is actually missing.

Develop Only When Necessary, Validate Before Adoption, and Update When Justified

How to Decide What a Prognostic Model Actually Needs

What Is Prognostic Model Development?

Prognostic model development involves constructing a mathematical function that combines predictor information to estimate the probability, timing, or other characteristics of a future outcome.

For example, researchers may develop a model estimating one-year mortality after diagnosis or the probability of academic withdrawal during the first year of enrollment.

Development requires defining the target population, prediction time, outcome, candidate predictors, and modeling strategy.

Researchers then estimate model parameters using development data and evaluate the model's apparent performance and optimism.

However, developing a model does not establish that it will perform adequately in other populations. Even a model with strong development performance may be overfitted or dependent on characteristics of the original setting.

The development stage should therefore include appropriate internal validation and a plan for subsequent independent evaluation.

What Is External Validation?

External validation evaluates an existing, previously developed model using data that were not used to construct or optimize it.

The purpose is to determine how accurately the fixed model predicts outcomes in a relevant population or setting.

Researchers should apply the original model, including its coefficients, intercept or baseline survival function, predictor definitions, and other necessary components, without first reestimating them using the validation outcomes.

They then assess performance using measures appropriate to the outcome and prediction horizon.

External validation may examine a different institution, geographic region, calendar period, clinical population, or independently recruited cohort.

However, independence alone does not guarantee meaningful external validation. The validation dataset should address a relevant question about the model's intended use or transportability.

What Is Model Updating?

Model updating modifies an existing prediction model to improve its performance in a particular population or setting.

The modification may be relatively limited, such as adjusting the model's intercept or baseline hazard, or more extensive, such as revising predictor coefficients.

Updating may be justified when the original model retains useful predictive information but systematically overestimates or underestimates risk.

For example, improvements in treatment may reduce the frequency of an outcome, causing a previously developed model to overpredict risk.

In that situation, recalibrating the model may be more appropriate than discarding its existing predictor structure and developing an entirely new model.

Updating must nevertheless be evaluated carefully because modifications based on limited data can introduce additional overfitting.

How Do Development, Validation, and Updating Compare?

These strategies may form a sequence. A model may first undergo external validation, followed by updating if its performance is inadequate. However, researchers should report the original validation results separately from performance after updating.

Why Should Researchers Search for Existing Models Before Developing One?

Prediction-model research has produced many models that are developed once and never independently validated.

Creating additional models without evaluating existing alternatives can contribute to research duplication and leave practitioners uncertain about which model is trustworthy.

Before development, researchers should search the literature for models addressing the same or closely related prediction task.

The comparison should consider the target population, outcome definition, prediction horizon, intended decision point, and predictor availability.

A model developed for five-year mortality cannot automatically be treated as suitable for predicting 30-day readmission. Similarly, a model requiring specialist laboratory measurements may be impractical in a primary care setting.

Where a credible existing model is available, independent validation may provide a more informative contribution than another development study.

What Makes an Existing Model Suitable for Validation?

The model should address a prediction task sufficiently similar to the intended application.

Researchers need access to its complete prediction equation or implementation, including all necessary parameters and predictor definitions.

For a logistic regression model, this generally includes the intercept and predictor coefficients. For a survival model, absolute risk prediction may also require baseline survival or cumulative hazard information at the relevant horizon.

Models published only as lists of significant predictors may be impossible to implement faithfully.

Researchers should also determine whether the predictors can be measured consistently and whether the model's original prediction time matches the intended decision.

When essential model information is unavailable, authors may need to be contacted before deciding that external validation is impossible.

What Is the Difference Between Internal and External Validation?

Internal validation examines the performance and stability of a model within the development data using procedures such as bootstrapping or cross-validation.

Its principal purpose is to assess optimism arising from fitting and evaluating a model using related observations.

External validation evaluates the fixed model using independent observations not involved in its development.

Both forms of validation are useful, but they address different questions.

Internal validation cannot establish transportability to another setting. External validation may reveal performance differences arising from changes in population characteristics, outcome frequency, measurement, or care practices.

The distinction between internal validation and simple train-test splitting is important when planning efficient model development.

Which Performance Measures Should Guide the Decision?

Two fundamental aspects of prognostic model performance are discrimination and calibration.

Discrimination describes how well a model distinguishes individuals with different outcome experiences. Calibration concerns agreement between predicted probabilities and observed outcome frequencies.

A model may discriminate adequately while systematically overpredicting risk.

Conversely, recalibration may improve absolute risk estimates without materially changing the ranking of individuals.

Researchers should also consider clinical or practical usefulness, particularly when predictions trigger decisions at specified thresholds.

The evaluation of discrimination, calibration, and clinical usefulness should correspond to the model's intended application.

When Is Recalibration Preferable to Developing a New Model?

Recalibration may be appropriate when a model retains useful discrimination but predicted probabilities are systematically too high or too low in the new population.

For example, suppose a model developed when an adverse outcome was relatively common is applied after improvements in treatment have reduced its incidence.

The model may still rank individuals reasonably well, but its predicted risks may be too high.

Updating the intercept or baseline risk may improve calibration without replacing the predictor effects.

However, intercept adjustment alone may be insufficient if the relationships between predictors and outcomes have also changed.

Researchers should examine the nature of miscalibration before choosing the updating strategy.

What Is Calibration-in-the-Large?

Calibration-in-the-large concerns whether the model's average predicted risk agrees with the overall observed outcome frequency.

For example, a model may predict an average risk of 25% in a population where the observed event frequency is 15%.

This indicates overall overprediction, although the exact interpretation depends on the outcome and validation method.

For binary logistic models, recalibration of the intercept can adjust the overall prediction level while retaining the original predictor coefficients.

For time-to-event models, updating baseline survival or hazard information may serve a related purpose.

However, overall calibration alone does not establish that predictions are accurate across the entire risk range.

What Is a Calibration Slope?

The calibration slope evaluates whether the spread of predicted risks is appropriate in the validation data.

In a commonly used logistic recalibration model, a slope of one corresponds to the ideal value under the specified framework.

A slope below one may indicate that predictions are too extreme, a pattern often associated with overfitting in development.

A slope above one may indicate predictions that are insufficiently differentiated, although interpretation depends on the setting and uncertainty.

Researchers should not treat a single slope estimate as a complete description of calibration.

Calibration plots and assessment of nonlinear deviations can reveal problems not captured by an overall intercept and slope.

When Should Predictor Coefficients Be Revised?

More extensive updating may be considered when recalibration alone does not adequately address performance problems.

For example, the association between a predictor and an outcome may differ in a new population because of changes in treatment, measurement procedures, or case mix.

Researchers may then consider revising selected coefficients or refitting the model.

However, coefficient revision uses additional outcome information and may introduce overfitting, especially when the updating dataset contains few events.

The degree of updating should therefore reflect the available sample size, the nature of the performance problem, and the scientific justification for changing the model.

More extensive revision is not automatically preferable to simpler recalibration.

When Is Developing a New Model Justified?

New development may be appropriate when no suitable existing model addresses the intended prediction task.

It may also be justified when existing models depend on unavailable predictors, use incompatible outcome definitions, or demonstrate persistently inadequate performance that cannot reasonably be corrected through updating.

Researchers may also investigate whether new information provides substantial predictive improvement beyond established models.

However, adding predictors or using a more complex algorithm does not automatically improve generalizability or usefulness.

A new model should be evaluated against relevant existing alternatives whenever feasible.

The justification should concern the scientific or practical gap rather than the novelty of the algorithm alone.

Why Does the Prediction Time Matter?

A model is designed to use information available at a specified decision point.

For example, a model predicting outcomes at diagnosis differs from one using treatment response measured several weeks later.

Even if both estimate the same outcome, their predictors and eligible populations may differ.

Researchers should therefore verify that an existing model's prediction time corresponds to the intended application.

Changing the prediction time may require a different model or a clearly defined dynamic prediction approach rather than simple recalibration.

The importance of defining time zero in prognostic studies extends directly to model validation and updating.

How Can Changes in Population Affect Model Performance?

Model performance may differ across populations because of changes in predictor distributions, outcome incidence, treatment practices, and measurement procedures.

A model developed in a specialist hospital may encounter different patients when applied in primary care.

Similarly, a model developed using data from one historical period may perform differently after substantial changes in clinical practice.

These differences do not necessarily mean the original model was poorly developed.

They may indicate that the model's predictions are not fully transportable to the new setting.

External validation should investigate these differences before researchers decide whether updating or replacement is necessary.

What Is Model Drift?

Model drift refers broadly to changes over time that affect the relationship between predictors, outcomes, and predictive performance.

Changes may occur in the distribution of predictor values, the outcome frequency, or the conditional relationship between predictors and outcomes.

For example, a model predicting hospital readmission may become poorly calibrated after changes in discharge practices or follow-up care.

In educational settings, a withdrawal-risk model may change in performance after institutions introduce new student-support policies.

Monitoring may identify when a previously validated model requires reassessment.

However, not every observed fluctuation indicates genuine drift. Sampling variation and changes in data quality must also be considered.

Can a Model Be Updated Using the Same Dataset Used for Validation?

Yes, but the resulting analysis must distinguish the original validation from the updating process.

Researchers may first evaluate the fixed model in an independent dataset and identify systematic miscalibration.

They may then use that dataset to recalibrate or revise the model.

However, performance measured in the same data after updating is no longer an independent external validation of the updated model.

The updating process should be evaluated using appropriate internal validation and, where possible, further independent data.

Reporting only the improved post-update performance can conceal the extent of modification and exaggerate confidence in transportability.

How Much Data Does Model Updating Require?

The amount of data required depends on the extent of updating and the precision needed.

Adjusting a single intercept generally requires fewer parameters than reestimating numerous predictor coefficients.

However, even simple recalibration requires adequate outcome information to estimate the adjustment reliably.

More extensive updating demands additional data and stronger safeguards against overfitting.

Researchers should not assume that a small validation dataset can support unrestricted model revision.

Sample size planning should account for the proposed updating method and the subsequent evaluation of model performance.

What Reporting and Appraisal Guidance Should Be Used?

TRIPOD+AI provides updated reporting guidance for clinical prediction models using regression or machine-learning methods, including development and evaluation studies.

PROBAST provides a framework for assessing risk of bias and applicability in prediction-model studies, while PROBAST+AI extends relevant appraisal guidance for contemporary prediction-model research.

Researchers should report the original model, validation population, performance measures, modifications made, and evaluation of the updated model.

Where possible, sufficient model information should be provided to permit independent implementation and further validation.

Transparent reporting is essential because the distinction between validation and updating can otherwise become obscured.

When Recalibration Is More Defensible Than Developing Another Model

The hypothetical findings suggest that recalibration may be a reasonable first response. They do not prove that intercept adjustment will fully correct calibration across all risk levels.

Developing a completely new model without investigating this simpler option could unnecessarily discard useful existing information.

Common Mistakes in Prognostic Model Development and Updating

How to Choose the Right Prognostic Model Research Strategy

Begin with a focused search for existing models addressing the intended population, outcome, prediction time, and horizon.

Evaluate whether these models are sufficiently reported, methodologically credible, and practical to implement.

Where an existing model is available, report its original performance before making modifications. This allows readers to distinguish model transportability from the effects of updating.

When updating is necessary, prefer the least extensive modification that adequately addresses the identified problem, subject to appropriate evaluation.

The most valuable contribution may be demonstrating that an existing model works, identifying where it fails, or showing how a justified modification improves performance.

Before Developing, Validating, or Updating a Prognostic Model

Frequently Asked Questions About Prognostic Model Development and Updating

The Research Gap Should Determine Whether to Develop, Validate, or Update

How to Cite This Guide

This guide is intended to be read, shared, and used in research, teaching, and academic work. If you draw on its ideas, explanations, or other content, please acknowledge the source by citing the guide. Doing so gives appropriate credit and helps your readers locate the original resource.