What Is Internal Validation, and Why Is a Simple Train-Test Split Often Inefficient?

Why Might Dividing Your Dataset Into Training and Testing Groups Be a Poor Validation Strategy?

You have 600 participants and want to develop a prediction model. Following a familiar machine-learning workflow, you randomly assign 70% of participants to training and 30% to testing.

The procedure seems straightforward. Develop the model using the training data, evaluate it using the testing data, and report the resulting accuracy.

But what happens if the outcome is uncommon? What if the training subset contains too few events for stable model development, or the test subset contains too few events to estimate performance precisely?

A single random split may protect against direct reuse of training outcomes in evaluation when conducted correctly. Yet it can also discard valuable development information and make conclusions depend heavily on one arbitrary partition of the data.

Internal validation offers more informative alternatives, particularly when observations and outcome events are limited.

Internal Validation Estimates Optimism Without Necessarily Reserving a Permanent Test Set

Why Resampling Often Provides a Better Internal Validation Strategy

What Is Internal Validation?

Internal validation evaluates aspects of prediction-model performance using the dataset available for development.

Its principal purpose is to assess optimism and instability arising when a model is fitted to a particular sample.

Rather than assuming that training performance represents performance in new observations, internal validation creates repeated opportunities to assess the model-development procedure using observations not involved in a particular fit.

Common approaches include bootstrap validation, k-fold cross-validation, and other suitable resampling procedures.

The relevant target is usually the performance of the complete development strategy, including preprocessing, predictor selection, and tuning, rather than only the final mathematical equation considered in isolation.

What Is a Train-Test Split?

A train-test split divides the available observations into separate subsets.

The training subset is used to fit the model, while the test subset is reserved for evaluation.

For example, a 70:30 split assigns 70% of observations to training and 30% to testing.

If the test outcomes are not used during development, the held-out subset can provide an internal estimate of predictive performance.

However, this estimate pertains to the particular model developed using the training subset and to the observations represented in the test subset.

It is not automatically an estimate of how a model subsequently refitted using all observations will perform.

Nor is it necessarily evidence of transportability to another setting.

Why Is a Single Random Split Often Inefficient?

Prediction-model development requires enough information to estimate relationships between predictors and outcomes.

When researchers reserve a substantial proportion of observations for testing, those observations no longer contribute to fitting the model evaluated by the split.

This can be particularly problematic when the outcome is uncommon or the model contains numerous candidate parameters.

The test subset may also be too small to estimate discrimination and calibration precisely.

Furthermore, the resulting performance estimate can depend on which participants happen to enter each subset.

Repeating the procedure with a different random seed may produce meaningfully different estimates, especially in smaller datasets.

Thus, a simple split may sacrifice development information without producing a sufficiently reliable evaluation.

How Much Information Is Lost in a Train-Test Split?

The amount depends on the split proportion and the number of available observations and events.

Only approximately 42 events would be available for fitting the model in this illustration, while the test subset would contain approximately 18 events for evaluating performance.

Such a small test-event count may produce substantial uncertainty in calibration and discrimination estimates.

The example does not establish a universal minimum event count. It illustrates how splitting can reduce the information available for both development and evaluation.

What Is Bootstrap Validation?

Bootstrap validation repeatedly draws samples with replacement from the original development dataset.

Each bootstrap sample ordinarily contains the same number of observations as the original dataset, although some original observations appear more than once and others are absent.

The model-development procedure is repeated in each bootstrap sample.

Performance is evaluated in the bootstrap sample and in the original dataset, allowing estimation of optimism associated with the development procedure.

Average optimism can then be used to adjust apparent performance estimates.

Bootstrap validation is particularly useful for many regression-based prediction models because it allows the development process to be assessed without permanently withholding a large subset of observations.

However, the procedure must be designed appropriately for the data structure and model-development strategy.

How Does Bootstrap Optimism Correction Work?

Suppose a model achieves an apparent AUC of 0.83 in its development dataset.

Researchers perform bootstrap validation, repeating the complete development procedure in each resample.

They calculate the difference between bootstrap-sample performance and the performance of the bootstrap-developed model in the original dataset.

The average difference estimates optimism.

The corrected estimate is not guaranteed to equal performance in an independent population.

It remains an estimate derived from the original data source and the assumptions of the resampling procedure.

What Is k-Fold Cross-Validation?

In k-fold cross-validation, the dataset is divided into k subsets or folds.

The model is developed using k minus one folds and evaluated in the remaining fold.

This process is repeated until every fold has served as the evaluation subset.

For example, five-fold cross-validation repeatedly fits models using approximately 80% of the observations and evaluates them using the remaining 20%.

Performance information is then combined across the folds.

The method provides an internal assessment of predictive performance without relying on only one training-testing partition.

However, its implementation must respect the outcome, dependence structure, and any model-selection procedures.

How Does Cross-Validation Compare With Bootstrapping?

No method is universally optimal. The choice depends on the available data, modeling algorithm, computational resources, and validation objective.

Does Cross-Validation Always Use Data More Efficiently?

Cross-validation often improves the use of limited data compared with a single random split, but efficiency depends on how the procedure is implemented.

For example, five-fold cross-validation trains each model on approximately 80% of observations, while ten-fold cross-validation uses approximately 90% for each fit.

However, smaller evaluation folds may contain relatively few events, and performance estimates can vary across resampling schemes.

Repeated cross-validation may help assess variation attributable to fold assignment.

Researchers should not assume that increasing the number of folds always produces better or more stable estimates.

The resampling design should be chosen according to the sample size, outcome structure, and analytical objective.

What Is Leave-One-Out Cross-Validation?

Leave-one-out cross-validation is an extreme form of cross-validation in which each iteration holds out one observation and fits the model using all remaining observations.

It can be computationally expensive for complex algorithms and does not necessarily provide the most stable performance estimate.

Its statistical properties differ from those of k-fold cross-validation, and it should not automatically be selected simply because each fitted model uses nearly all observations.

Researchers should evaluate whether the additional computational cost and characteristics of the estimator suit the prediction task.

What Is Nested Cross-Validation?

Nested cross-validation separates model tuning from final internal performance evaluation.

An inner resampling procedure is used to select hyperparameters or compare candidate modeling configurations.

An outer resampling procedure evaluates the development strategy, including the tuning decisions made within the inner procedure.

This separation reduces the risk of reporting overly optimistic performance after selecting the best configuration using the same evaluation results.

Nested cross-validation can be particularly useful when researchers compare numerous algorithms or tune complex machine-learning models.

However, it may be computationally demanding, and the evaluation must still reflect the intended data structure.

Why Must Feature Selection Be Repeated During Resampling?

Feature selection uses information to determine which predictors enter the model.

If researchers select predictors using the entire dataset before cross-validation, evaluation-fold outcomes may influence the selected feature set.

The subsequent cross-validation no longer provides an appropriately separated assessment of the full development procedure.

For example, screening 100 candidate predictors using univariable associations across all observations before creating folds can introduce substantial leakage.

Where selection depends on the observed outcomes, it should be repeated within the training portion of each resampling iteration.

The same principle applies to tuning procedures that use outcome information.

How Should Missing-Data Imputation Be Handled?

Missing-data procedures can also influence validation.

For example, estimating imputation parameters using all observations before creating cross-validation folds may allow evaluation data to influence the development procedure.

For a deployment-oriented prediction pipeline, imputation procedures should generally be learned using training observations and applied to held-out observations without using their outcomes.

Multiple imputation introduces additional considerations because it aims to account for uncertainty about missing values, and its implementation should be consistent with the intended validation estimand.

Researchers should document how missing predictors are handled during development and how the same procedure would operate for new individuals.

Imputation should not be treated as an inconsequential preliminary step.

Does Standardization Need to Occur Within Training Folds?

When standardization parameters are estimated from data, they should generally be learned from the training observations and applied to the evaluation observations.

For example, a predictor's mean and standard deviation may be calculated in the training fold and then used to standardize that fold and its corresponding evaluation fold.

Calculating these quantities using the entire dataset beforehand introduces information from evaluation observations into preprocessing.

The practical effect may be small in some settings, but the correct validation principle is to reproduce the procedure that would be available when the model is applied to new data.

Researchers should apply this principle to normalization, dimensionality reduction, and other learned transformations.

Why Does the Unit of Resampling Matter?

Many datasets contain observations that are not independent.

Examples include repeated measurements from the same patient, multiple assessments from one student, or individuals nested within institutions.

If records from the same person appear in both training and evaluation folds, the model may benefit from information about that person that would not be available when predicting outcomes for a genuinely new individual.

Similarly, random splitting across institutions may conceal performance differences between settings.

Grouped cross-validation or other cluster-aware methods may be appropriate depending on the intended application.

The resampling unit should correspond to the level at which independent prediction performance is needed.

When Is Temporal Validation Preferable to Random Resampling?

When the intended use involves predicting outcomes in future calendar periods, evaluating a model on later observations may provide useful information about temporal performance.

For example, a student withdrawal model developed using earlier enrollment cohorts may be evaluated using a subsequent cohort.

Such evaluation can reveal changes in enrollment characteristics, institutional policies, or outcome frequency.

However, temporal validation is not interchangeable with random internal resampling.

It addresses performance across time, while random resampling primarily evaluates performance within the original data-generating context.

Researchers should consider both when the intended deployment environment may change.

Does Internal Validation Establish Calibration?

Internal validation can assess calibration, but calibration must be evaluated explicitly.

A model may achieve reasonable cross-validated discrimination while systematically overestimating or underestimating outcome probabilities.

Researchers should examine measures such as calibration-in-the-large, calibration slope, and calibration plots where appropriate.

For time-to-event outcomes, calibration assessment must account for the prediction horizon and censoring.

The choice of internal validation method should support the performance measures relevant to the intended decision.

Evaluating only classification accuracy or AUC is often insufficient for risk prediction.

What If the Outcome Is Rare?

Rare outcomes create additional difficulties because relatively few events may be available in individual folds or test subsets.

Some partitions may contain very few events, making discrimination and calibration estimates unstable or even undefined for particular measures.

Stratified fold allocation may help distribute events across folds, although it does not create additional outcome information.

Researchers should also consider whether the overall sample size is sufficient for the proposed model complexity.

Resampling cannot compensate completely for inadequate event information.

The relevant considerations are discussed in how many outcome events prognostic studies need.

Can Resampling Eliminate Overfitting?

No. Internal validation can estimate optimism and reveal unstable performance, but it does not automatically prevent overfitting.

Overfitting may be reduced through appropriate model complexity, penalization, shrinkage, and sufficient development data.

Resampling can help evaluate the consequences of these decisions and guide model-development strategies.

However, a severely underpowered dataset may remain inadequate even when bootstrap or cross-validation procedures are technically correct.

Researchers should distinguish methods for assessing overfitting from methods for reducing it.

Does Internal Validation Replace External Validation?

No. Internal validation assesses performance within the available development data source.

External validation evaluates a fixed model using independent observations relevant to its intended application.

For example, a model may perform consistently across random folds from one hospital while performing poorly in another hospital with different patients or measurement procedures.

Likewise, a model developed at one university may not retain its calibration after being introduced into an institution with different admission policies and student-support systems.

Researchers should distinguish external validation and its contribution to model generalizability from internal optimism assessment.

How Should Internal Validation Be Reported?

Researchers should identify the validation method, number of bootstrap repetitions or cross-validation folds, and the procedures used for preprocessing, feature selection, and tuning.

They should explain whether observations were clustered or repeated and how this structure was addressed.

Apparent performance and internally validated estimates should be distinguished.

For cross-validation, the method used to combine fold-level predictions or performance measures should be specified.

For bootstrap validation, the optimism-correction procedure should be described.

TRIPOD+AI provides reporting guidance relevant to these decisions in clinical prediction-model studies.

Why a 70:30 Split May Provide Less Useful Information Than Resampling

The example does not prove that bootstrapping is always superior to a random split. It illustrates why reserving a large proportion of a limited dataset may be inefficient.

When substantial independent data are available, a separate evaluation dataset can be valuable. The methodological priority is to match the validation strategy to the research question, available information, and intended application.

Common Mistakes in Internal Validation and Data Splitting

How to Choose an Efficient Internal Validation Method

Begin by assessing the number of participants, outcome events, candidate predictor parameters, and dependence structure.

Then identify whether the objective is to estimate optimism, compare modeling strategies, tune hyperparameters, or evaluate performance across meaningful clusters.

Document every step that learns information from the data, including imputation, scaling, feature selection, tuning, and model fitting.

Ensure that the resampling procedure evaluates the complete development strategy and does not allow evaluation outcomes to influence training decisions.

Internal validation should be designed to produce a credible assessment of model optimism while using the available data appropriately.

Before Choosing an Internal Validation Strategy

Frequently Asked Questions About Internal Validation and Train-Test Splits

Internal Validation Should Assess Optimism Without Unnecessarily Wasting Data

How to Cite This Guide

This guide is intended to be read, shared, and used in research, teaching, and academic work. If you draw on its ideas, explanations, or other content, please acknowledge the source by citing the guide. Doing so gives appropriate credit and helps your readers locate the original resource.