03 · What You Need to Know
How Development Data Can Support Validation Without Proving Generalizability
What Does It Mean to Develop and Validate a Model?
Model development involves constructing a prediction function from observed relationships between predictors and outcomes.
For a logistic regression model, development typically includes estimating an intercept and predictor coefficients. For a machine-learning model, it may include learning parameters, choosing hyperparameters, selecting features, and determining preprocessing procedures.
Validation evaluates how well the resulting prediction function performs when applied to observations that were not used to fit or optimize that particular model.
However, validation is not a single interchangeable activity. Internal validation and external validation address different forms of uncertainty.
Researchers should therefore distinguish prediction model development, validation, and updating before deciding how the available observations will be used.
What Is Apparent Performance?
Apparent performance is calculated by applying a fitted model to the same observations used to develop it.
For example, researchers may fit a logistic regression model to 800 participants and calculate its area under the receiver operating characteristic curve using those same 800 participants.
This result describes how well the model fits and predicts within the development sample. It does not provide an independent assessment of how the model will perform in new observations.
Apparent performance is often optimistic because model fitting can capture sample-specific patterns in addition to reproducible predictor-outcome relationships.
The extent of optimism depends on model complexity, sample size, outcome frequency, predictor selection, tuning, and other development decisions.
Apparent performance is useful to report, but it should be identified accurately rather than presented as validation.
Why Is Evaluating a Model on Its Training Data Problematic?
A model is constructed to fit relationships observed in its development data. Some of those relationships reflect genuine patterns, while others arise from random variation.
Flexible modeling methods may capture these chance patterns particularly readily when the available information is limited relative to model complexity.
When performance is evaluated using the same observations, the model benefits from having already encountered their outcomes during development.
Its discrimination, calibration, or prediction error may therefore appear more favorable than would be observed in new participants.
This is a manifestation of overfitting.
Overfitting is not restricted to machine learning. Conventional regression models can also overfit, particularly when many candidate parameters are considered or extensive data-driven selection is performed.
What Is Internal Validation?
Internal validation uses the development dataset to estimate how well the modeling procedure may perform beyond the observations used for fitting.
It commonly relies on resampling or data partitioning to create repeated opportunities for model development and evaluation.
Appropriate methods include bootstrapping and cross-validation. A single random training-testing split is another form of internal assessment, although it can be inefficient and unstable in smaller datasets.
Internal validation is particularly valuable when independent external data are not yet available.
However, its conclusions remain conditional on the characteristics and data-generating processes represented in the original dataset.
It cannot establish that predictor relationships, measurement procedures, or baseline risks will remain unchanged elsewhere.
What Is Bootstrap Internal Validation?
Bootstrap validation repeatedly draws samples with replacement from the development dataset.
Within each bootstrap sample, researchers repeat the model-development procedure and evaluate the fitted model's performance both in that sample and in the original data.
The difference between these performance estimates can be used to estimate optimism associated with the development process.
Average optimism across bootstrap repetitions can then be used to adjust apparent performance estimates.
For example, the optimism-corrected AUC may be calculated by subtracting estimated average optimism from the apparent AUC.
The bootstrap procedure must reproduce relevant development decisions, including predictor selection and tuning, if the resulting optimism estimate is intended to reflect those decisions.
Simply bootstrapping a final model after ignoring an extensive selection process may underestimate the uncertainty introduced during development.
How Is Bootstrap Optimism Correction Calculated?
For error measures where smaller values indicate better performance, the direction of the correction must be handled appropriately.
Bootstrap validation also involves assumptions about the representativeness of the development sample. It does not automatically correct systematic selection bias or measurement error.
What Is Cross-Validation?
Cross-validation divides the available observations into subsets, commonly called folds.
In k-fold cross-validation, the model is fitted using k minus one folds and evaluated in the remaining fold. The procedure is repeated so that each fold serves as the evaluation subset.
The results are then combined to estimate performance under the chosen resampling procedure.
Cross-validation can be useful for estimating predictive performance and selecting model hyperparameters.
However, using the same cross-validation results both to optimize a model and to report its final performance can introduce selection-related optimism.
Nested cross-validation or another appropriately separated evaluation strategy may be necessary when extensive tuning or model comparison is performed.
Why Must Preprocessing Be Included Within Validation?
Data preprocessing can use information from the dataset even when it does not involve fitting the final prediction equation.
Examples include missing-data imputation, variable selection, scaling, transformation, dimensionality reduction, and hyperparameter tuning.
If preprocessing is estimated using all observations before cross-validation, information from evaluation folds may influence the models being assessed.
For example, selecting predictors using outcome associations calculated from the complete dataset before dividing it into folds allows validation outcomes to influence feature selection.
This is a form of information leakage.
Where preprocessing involves learned parameters or outcome information, it should generally be performed within the training portion of each resampling iteration, with the learned procedure applied to the corresponding evaluation observations.
The precise implementation depends on the preprocessing method and the prediction task.
Is a Random Train-Test Split a Form of External Validation?
Ordinarily, no. A random split divides observations from the same source dataset into development and evaluation subsets.
Although the test observations are not used directly to fit the model when the procedure is conducted correctly, both subsets generally represent the same underlying population and measurement processes.
The test set therefore provides an internal assessment of performance within that source.
It does not necessarily reveal problems that arise when the model is applied to another hospital, university, geographic region, or calendar period.
The distinction is important because independent observations are not automatically equivalent to independent settings.
Why Can a Single Random Split Be Inefficient?
A random split reserves part of the available dataset for testing, reducing the number of observations and outcome events available for model development.
For example, a 70:30 split leaves only 70% of observations for fitting the model.
If the outcome is uncommon, the test subset may contain relatively few events, producing imprecise performance estimates.
Results can also depend substantially on the particular random allocation of participants.
Resampling approaches such as bootstrapping can often make more efficient use of limited development data.
The choice should nevertheless reflect dataset size, computational requirements, clustering, temporal structure, and the modeling objective.
The methodological issues are examined further in why a simple train-test split is often inefficient.
Does Internal Validation Require a Separate Validation Sample?
Not necessarily. Bootstrap internal validation repeatedly uses observations from the development dataset without requiring a permanently reserved validation subset.
Cross-validation likewise rotates evaluation subsets across the available observations.
These procedures allow the same overall dataset to support both model development and internal validation while maintaining appropriate separation within individual resampling iterations.
However, the final model is generally fitted using the available development data after the modeling procedure has been established.
The internally validated performance estimate describes the expected performance of that development strategy under the relevant assumptions, rather than constituting an untouched external test of the final fitted model.
When Is External Validation Necessary?
External validation is especially important before a prediction model is adopted broadly or used for consequential decisions.
It evaluates the fixed model using independent observations not involved in its development or optimization.
The validation population should be relevant to the model's intended application.
For example, a model developed using patients from a specialist hospital may require evaluation in primary care if primary care is its intended deployment setting.
A student withdrawal model developed at one institution may require independent assessment before use at another institution with different enrollment patterns or support systems.
External validation can identify differences in calibration, discrimination, and applicability that internal validation cannot adequately establish.
Can a Model Be Externally Validated in the Same Institution?
Yes. A model developed using an earlier cohort may be evaluated in a later independent cohort from the same institution.
This is commonly described as temporal validation.
It can reveal changes in outcome frequency, population characteristics, treatment practices, or institutional procedures.
However, temporal validation within one institution may provide less evidence about geographic transportability than evaluation across genuinely different settings.
Researchers should describe which dimension of external validity the design is intended to assess.
What Happens if the Model Is Modified During External Validation?
External validation should initially evaluate the fixed original model.
If researchers change coefficients, recalibrate probabilities, select new predictors, or optimize thresholds using the validation outcomes, the analysis becomes a form of model updating or development.
The original validation results should be reported before modification.
Performance after updating in those same observations is not independent external validation of the revised model.
The updated model requires appropriate internal assessment of the updating process and, where feasible, further independent validation.
This distinction prevents researchers from interpreting improved apparent performance as evidence that the modification will generalize.
What Performance Measures Should Internal Validation Examine?
Validation should assess performance measures relevant to the intended prediction task.
Discrimination concerns the ability to distinguish individuals with different outcomes. Calibration concerns agreement between predicted probabilities and observed outcome frequencies.
For binary outcomes, discrimination may be summarized using AUC. Calibration can be assessed through plots, calibration-in-the-large, and calibration slope.
For continuous outcomes, relevant measures may include prediction error and agreement between predicted and observed values.
Time-to-event outcomes require methods appropriate to censoring, prediction horizons, and competing events where relevant.
Researchers should avoid evaluating only discrimination when the model is intended to support decisions based on absolute risk.
Can Internal Validation Correct a Poorly Designed Dataset?
No. Internal validation addresses aspects of model overfitting and performance estimation within the available data source.
It cannot automatically correct selection bias, systematic outcome misclassification, inappropriate predictor timing, or inadequate representation of the intended population.
For example, a model developed using only advanced disease cases and healthy controls may show stable cross-validated performance while remaining poorly applicable to patients with uncertain diagnoses.
Likewise, a model containing information unavailable at the intended prediction time may remain invalid for that application even if cross-validation is implemented correctly.
Study design and data validity must therefore be evaluated independently of the resampling procedure.
How Do Related Observations Affect Validation?
Ordinary random resampling may be inappropriate when observations are clustered or dependent.
Examples include repeated measurements from the same patient, multiple records from one student, or observations nested within institutions.
If records from the same individual appear in both training and evaluation subsets, the model may benefit from information that would not be available for a genuinely new individual.
Researchers should select resampling units and partitioning strategies that reflect the intended prediction task.
Grouped cross-validation, temporal partitioning, or internal-external validation across clusters may be appropriate in particular circumstances.
The relevant independence structure should be established before validation begins.
How Much Data Is Needed for Development and Validation?
Sample size requirements depend on the outcome, candidate predictor parameters, anticipated predictive performance, and intended validation precision.
For model development, researchers should consider overfitting and the stability of predicted risks.
For external validation, sample size planning should focus on sufficiently precise estimates of calibration and discrimination.
Using the same overall dataset for internal validation does not remove the need for adequate development information.
The principles governing outcome-event requirements in prognostic research remain relevant.
How Should Development and Validation Be Reported?
Researchers should identify the data used for model fitting, feature selection, preprocessing, tuning, and evaluation.
They should specify the internal validation method, including the resampling structure and whether the complete development process was repeated.
Apparent and optimism-corrected performance should be distinguished.
If external validation was conducted, the independent dataset and its relationship to the development population should be described.
TRIPOD+AI provides reporting guidance for clinical prediction-model research using regression or machine-learning methods.
Transparent reporting allows readers to determine whether the study supports internal performance assessment, external transportability, or both.
