Model Development vs. Validation vs. Updating: Which Prediction Model Study Are You Doing?

Are You Creating a Model, Testing an Existing One, or Changing It?

You have obtained a dataset containing predictors and outcomes. You identify an existing prediction model, apply it to your participants, and discover that its predictions are systematically too high. You adjust its intercept, reestimate several coefficients, and report improved performance.

Have you externally validated the model, updated it, or developed a new one?

The answer matters because these activities use data differently and support different scientific claims. A model evaluated without modification provides evidence about its transportability. A model modified using the evaluation data becomes a revised prediction function whose performance requires additional assessment.

Researchers should therefore identify the actual modeling activity before selecting the analysis or describing the study in a manuscript.

Classify the Study According to What You Do With the Model and Data

How to Distinguish Prediction Model Development, Validation, and Updating

What Is Prediction Model Development?

Development involves constructing a prediction model using a dataset containing candidate predictors and observed outcomes.

Researchers define the target population, prediction time, outcome, and intended application. They then select a modeling strategy and estimate the parameters required to generate predictions.

For a logistic regression model, these parameters include the intercept and predictor coefficients. For a time-to-event model, absolute risk estimation may additionally require a baseline survival or cumulative hazard function.

Machine-learning approaches may involve learned parameters, transformations, tuning choices, and preprocessing procedures.

All components required to generate predictions form part of the developed model.

Development should be accompanied by appropriate internal validation because performance measured in the development data is often optimistic.

What Is Apparent Model Performance?

Apparent performance is the performance obtained when a model is evaluated using the same observations involved in fitting or selecting it.

For example, researchers may fit a logistic regression model to 1,000 participants and calculate its AUC using those same participants.

The result describes performance in the development data, but it may be optimistic because the model was constructed to fit those observations.

Apparent performance is not equivalent to independent validation.

The extent of optimism depends on model complexity, available information, predictor selection, tuning, and other aspects of the development process.

Researchers should report apparent performance accurately while recognizing its limitations.

What Is Internal Validation?

Internal validation evaluates how much model performance may be affected by overfitting within the development dataset.

Common approaches include bootstrapping and appropriately implemented cross-validation.

These methods repeatedly separate aspects of model fitting and evaluation to estimate performance beyond the particular observations used for fitting.

For example, bootstrap validation can estimate optimism by comparing performance in bootstrap samples with performance when the fitted models are applied to the original sample.

However, the entire development process should be incorporated, including predictor selection and tuning where applicable.

Internal validation does not establish that a model will perform adequately in a substantially different population.

The distinction between internal validation and a simple train-test split is important when development data are limited.

What Is External Validation?

External validation assesses a previously developed model using independent data not involved in its development or optimization.

The original prediction function should be applied without using the validation outcomes to change its parameters beforehand.

Researchers then compare predictions with observed outcomes using appropriate performance measures.

External validation can investigate whether a model performs adequately in another institution, geographic location, calendar period, or relevant population.

However, a different dataset is not automatically informative if it does not represent the intended prediction task.

The validation population, predictor definitions, prediction time, and outcome horizon should be evaluated for compatibility with the original model.

How Do the Main Study Types Compare?

A single research project may include several rows of this table. However, each activity should have its own objective, analysis, and interpretation.

What Makes Validation Independent?

Independence means that the evaluation outcomes have not been used to construct, select, tune, or modify the model being evaluated.

For example, if researchers repeatedly test different model configurations against the same validation dataset and select the best-performing version, that dataset has contributed to model development.

The final reported performance is no longer an untouched independent evaluation.

Similarly, selecting a new prediction threshold using validation outcomes changes the evaluated decision rule.

Researchers should distinguish the original fixed-model assessment from subsequent optimization.

This principle applies to conventional regression and machine-learning systems alike.

Does External Validation Require a Different Institution?

No. External validation concerns independence and the scientific question about performance beyond development conditions.

A later cohort from the same institution may provide temporal validation, particularly when practice patterns or outcome frequencies have changed.

A geographically distinct institution may provide evidence about transportability across settings.

However, merely using different records from the same underlying development population does not necessarily provide strong evidence of broader applicability.

Researchers should describe the relationship between development and validation populations and explain which aspect of generalizability is being evaluated.

What Is Model Recalibration?

Recalibration modifies a model to improve agreement between predicted probabilities and observed outcomes.

For example, a model developed in a population with relatively high event incidence may systematically overpredict risk in a population with lower incidence.

If the original predictor effects remain suitable, adjusting the model's intercept or baseline risk may improve calibration.

More extensive recalibration may also modify the calibration slope.

Recalibration does not necessarily require changing the predictors or completely rebuilding the model.

However, the modification uses outcome information and therefore creates an updated model requiring appropriate evaluation.

How Does Logistic Recalibration Work?

For a logistic prediction model, recalibration may involve estimating an intercept and slope using the original model's linear predictor.

Recalibration procedures differ for survival models and other outcome structures. Researchers should use methods appropriate to the original model rather than applying logistic recalibration indiscriminately.

What Is Model Revision?

Model revision changes predictor coefficients, predictor definitions, or the structure of an existing model.

For example, researchers may discover that an established predictor has a different relationship with the outcome in the new population.

They may then consider revising that coefficient or adding a new predictor that provides important additional information.

Such changes can improve performance but also increase the risk of overfitting.

The more parameters researchers reestimate, the more outcome information is generally required.

Revision should therefore follow a clear rationale and should not be undertaken solely because it improves apparent performance in one dataset.

When Does Updating Become New Development?

The boundary is not always sharply defined.

A model that retains its original predictors and coefficients but updates an intercept is clearly a recalibrated version of the existing model.

A model that reestimates most coefficients, adds numerous predictors, and substantially changes its functional form may resemble new development.

Researchers should describe the actual modifications rather than relying only on a broad label.

The relevant methodological questions concern how much information was borrowed from the original model, which parameters were reestimated, and how the revised prediction function was evaluated.

Why Should Validation Precede Updating?

Evaluating the original model first establishes how it performs before modification.

This provides evidence about transportability and identifies the nature of any performance problem.

For example, a model may retain good discrimination but show systematic overprediction. That pattern suggests a different updating strategy from a model with poor discrimination and severe calibration problems.

If researchers immediately modify the model and report only improved performance, readers cannot determine whether the original model was already adequate.

Reporting the initial validation separately also helps distinguish the contribution of updating from the performance of the existing model.

Can the Same Dataset Be Used for Validation and Updating?

Yes, provided the sequence and interpretation are transparent.

Researchers may first apply the fixed original model and report its performance in the independent dataset.

They may then use that dataset to recalibrate or revise the model.

However, performance of the updated model measured in those same observations is apparent performance for the updated version.

Internal validation of the updating procedure and further independent validation may be needed to assess whether improvements persist.

Researchers should avoid describing the post-update results as though the updated model had undergone independent external validation.

How Does Sample Size Planning Differ?

Model development, external validation, and updating have different sample size objectives.

Development requires enough information to estimate model parameters while limiting overfitting and obtaining sufficiently precise predictions.

External validation requires enough information to estimate calibration and discrimination with useful precision.

Updating requires sufficient outcome information for the number and type of parameters being modified.

For example, intercept-only recalibration generally involves fewer newly estimated parameters than complete coefficient revision.

However, neither activity should be assumed adequate merely because a conventional events-per-variable rule is satisfied.

Researchers should apply sample size principles appropriate to prognostic model objectives.

Why Is Calibration Important in Model Validation?

Calibration evaluates agreement between predicted probabilities and observed outcome frequencies.

A model may have good discrimination while systematically overpredicting or underpredicting risk.

For example, it may assign an average predicted risk of 30% in a population where the observed outcome frequency is 15%.

Such disagreement can affect decisions based on absolute risk thresholds.

Researchers should examine calibration-in-the-large, calibration slope, and graphical calibration where appropriate.

A single performance statistic rarely provides a complete account of model reliability.

Why Is Discrimination Not Enough?

Discrimination measures how well a model distinguishes individuals with different outcomes.

For binary outcomes, the AUC is commonly used to summarize ranking performance.

However, a model can rank individuals reasonably well while producing inaccurate absolute probabilities.

For example, reducing every predicted probability by an inappropriate amount may preserve much of the ranking while damaging calibration.

Researchers should therefore evaluate calibration, discrimination, and practical usefulness rather than choosing a model solely because its AUC is high.

What Is Model Transportability?

Transportability concerns whether a prediction model performs adequately when applied beyond its original development conditions.

Changes in patient characteristics, institutional practices, outcome frequency, measurement procedures, or treatment may affect performance.

For example, a model developed in a specialist hospital may encounter a different distribution of risk when applied in primary care.

External validation helps assess whether the original model remains reliable in the intended setting.

When transportability is limited, recalibration or more extensive updating may be considered.

However, researchers should investigate why performance differs rather than assuming that all differences require complete model replacement.

What Is Temporal Validation?

Temporal validation evaluates a model using observations collected during a different period from its development data.

It can be particularly useful when treatment practices, institutional policies, or population characteristics change over time.

For example, a model developed using records from 2015 to 2020 may be evaluated in an independent cohort from 2023 to 2025.

However, temporal separation alone does not guarantee broad generalizability.

The population and data-generating process may remain highly similar, or important differences may require careful interpretation.

Researchers should report the temporal relationship and assess whether the validation period is relevant to intended use.

What Is Internal-External Cross-Validation?

Internal-external cross-validation is a strategy that can be used when development data contain multiple meaningful clusters, such as hospitals, regions, or studies.

Researchers repeatedly develop a model using several clusters and evaluate it in a held-out cluster.

This provides information about performance across settings and may help identify heterogeneity in model transportability.

However, it differs from validating a final fixed model in a wholly independent external dataset.

The development and validation procedures must be carefully specified to avoid information leakage between clusters.

The method is particularly useful when researchers want to investigate whether predictive performance varies across participating populations.

When Should an Existing Model Be Replaced?

Replacement may be justified when existing models do not address the intended prediction task or perform inadequately despite reasonable updating options.

For example, a model may require predictors that are unavailable in the intended setting, or its outcome definition may differ fundamentally from the desired decision.

Alternatively, substantial changes in the underlying process may make the original predictor structure unsuitable.

However, replacement should not be based solely on the availability of a newer algorithm or a modest improvement in apparent development performance.

Researchers should compare candidate models using appropriate validation and assess whether any improvement is meaningful for the intended use.

How Should Combined Development and Validation Studies Be Described?

Researchers should describe each analytical stage explicitly.

For example, a study might develop a model in one cohort, perform internal validation using bootstrapping, and externally validate the final model in a separate cohort.

Another study might externally validate an existing model, recalibrate it, and evaluate the updated version.

These designs should not be collapsed into vague descriptions such as model validation when substantial development or modification occurred.

Reporting should identify which data were used for each activity and whether any observations contributed to more than one stage.

Transparency about data use is essential for interpreting performance estimates.

Which Reporting and Appraisal Frameworks Apply?

TRIPOD+AI provides reporting guidance for clinical prediction models developed or evaluated using regression and machine-learning methods.

PROBAST provides a framework for assessing risk of bias and applicability in prediction-model studies, with updated guidance addressing contemporary modeling approaches.

These frameworks emphasize participant selection, predictor and outcome definitions, sample size, modeling methods, and performance evaluation.

Researchers should also report the complete model specification and any modifications made during updating.

Appropriate reporting allows independent researchers to reproduce predictions and assess the validity of the claimed development, validation, or updating activity.

How One Dataset Can Support Validation Followed by Model Updating

The example demonstrates why the same dataset can contribute to more than one modeling activity without making those activities equivalent.

The critical distinction is whether the outcome information was used only to evaluate the original model or was also used to change the prediction function.

Common Misconceptions About Prediction Model Study Types

How to Identify the Prediction Model Study You Are Conducting

Begin by asking whether the prediction function already exists and whether you intend to change it.

If the model does not exist, the activity is development. If it exists and you evaluate it without modification, the activity is validation. If you use outcome information to change it, the activity includes updating.

For the protocol, specify the original model, the data used for each stage, the intended performance measures, and any planned updating procedures.

When several models are compared, apply a consistent evaluation framework and distinguish prespecified comparisons from exploratory optimization.

Accurate classification of the study type is essential because it determines which results can be interpreted as independent evidence and which require further validation.

Before Labeling a Prediction Model Study

Frequently Asked Questions About Prediction Model Study Types

The Study Type Depends on Whether the Prediction Function Is Created, Tested, or Changed

How to Cite This Guide

This guide is intended to be read, shared, and used in research, teaching, and academic work. If you draw on its ideas, explanations, or other content, please acknowledge the source by citing the guide. Doing so gives appropriate credit and helps your readers locate the original resource.