03 · What You Need to Know
Planning an Adequately Sized External Validation Study
What Is External Validation of a Prediction Model?
External validation evaluates an existing prediction model using data that were not used to develop or tune the model.
The validation population may differ from the development population in location, institution, time period, participant characteristics, or another relevant dimension.
For example, a dropout prediction model developed using students enrolled in one university might be evaluated using students from another university.
Alternatively, a model developed using records from one academic year might be validated using a later cohort.
The purpose is to examine how well the original model performs in a relevant new setting without using the validation outcomes to refit the model before assessing its original performance.
External validation provides evidence about transportability, but one successful validation study does not guarantee that the model will perform equally well in every future population.
Why Does External Validation Need Its Own Sample-Size Calculation?
Prediction model development and external validation have different statistical objectives.
Development requires sufficient information to estimate model parameters while limiting overfitting.
External validation primarily requires sufficient information to estimate the performance of an already specified model.
The relevant sample-size calculation should therefore focus on the precision of validation performance measures rather than the number of coefficients in the original model alone.
A complex development model may require substantial data to estimate its parameters, while its external validation requirement depends on the performance characteristics being evaluated and the target population.
Conversely, a simple model can require a large validation sample when the outcome is rare or when narrow confidence intervals are needed.
What Performance Measures Should External Validation Examine?
The relevant performance measures depend on the outcome type and intended use of the model.
For binary prediction models, calibration and discrimination are particularly important. Overall prediction error and performance at clinically or practically relevant decision thresholds may also matter.
| Performance measure |
What it evaluates |
Sample-size implication |
| Calibration-in-the-large |
Whether predictions are systematically too high or too low overall. |
Requires sufficient information to estimate overall calibration with useful precision. |
| Calibration slope |
Whether predicted risks are too extreme or insufficiently variable. |
Precision depends on outcome information and the distribution of predicted risks. |
| Calibration curve |
How observed outcome frequencies correspond to predicted risks across their range. |
Reliable estimation may require substantial data, especially in sparsely populated risk regions. |
| Discrimination |
How well predictions distinguish individuals with different outcomes. |
Precision depends on the event and nonevent counts and anticipated discrimination. |
| Overall prediction error |
How closely predictions correspond to observed outcomes. |
Requires a variance calculation appropriate to the selected performance measure. |
Researchers should prespecify the performance measures that matter for the model's intended application.
Planning solely around a convenient metric can leave other important aspects of performance inadequately estimated.
Why Is Calibration Important in External Validation?
Calibration concerns agreement between predicted outcomes and observed outcomes.
For example, among students assigned an estimated dropout risk of approximately 20%, a well-calibrated model would predict event frequencies broadly consistent with that risk, allowing for sampling variation.
A model may distinguish higher-risk from lower-risk students while systematically overestimating or underestimating their actual probabilities.
This matters when predictions are used to allocate resources or make decisions based on numerical risk thresholds.
Calibration should therefore be evaluated alongside discrimination rather than treated as an optional secondary consideration.
What Is Calibration-in-the-Large?
Calibration-in-the-large assesses whether predicted risks are systematically too high or too low overall.
In logistic calibration models, this is often represented by a calibration intercept estimated while holding the original prediction linear predictor fixed as an offset.
An intercept of zero indicates no systematic shift in the overall predicted risk on the logit scale under the specified calibration model.
A negative intercept suggests that the original model tends to overpredict the outcome overall, while a positive intercept suggests underprediction.
The precision of this estimate depends on the number of observations, the event distribution, and the predicted probabilities.
External validation sample-size planning can explicitly target an acceptable confidence-interval width for calibration-in-the-large.
What Is the Calibration Slope?
The calibration slope assesses how the spread of the model's predicted risks corresponds to the observed outcome relationships.
For a logistic prediction model, a common calibration analysis fits the observed binary outcome against the original model's linear predictor.
Under ideal calibration, the slope is one.
A slope below one often indicates that the model's predictions are too extreme in the validation population, although interpretation depends on the broader calibration pattern.
A slope above one may indicate that the predictions are insufficiently spread out.
The sample needed to estimate this slope precisely depends partly on the distribution of predicted risks and the number of outcome events.
A validation dataset with little variation in predicted risk may provide limited information about the slope even when its total sample is substantial.
What Is Discrimination?
Discrimination describes how well a prediction model distinguishes individuals with different observed outcomes.
For binary outcomes, a commonly used measure is the concordance statistic or C-statistic, equivalent to the area under the receiver operating characteristic curve under the usual definitions.
The C-statistic can be interpreted as the probability that a randomly selected event observation receives a higher predicted risk than a randomly selected nonevent observation, with an appropriate convention for ties.
A value of 0.50 corresponds to chance-level ranking under the usual interpretation, while a value of one indicates perfect separation of observed outcome categories.
However, good discrimination does not guarantee accurate predicted probabilities.
A model may achieve a high C-statistic while being poorly calibrated in the validation population.
How Does Outcome Prevalence Affect External Validation Sample Size?
For binary outcomes, the total number of validation observations does not fully describe the information available.
Suppose the outcome occurs in only 5% of participants. A validation sample of 1,000 individuals would contain approximately 50 events on average.
If the outcome occurs in 25%, the same total sample would contain approximately 250 events.
The precision of several validation statistics depends on the distribution of events and nonevents, not simply the number of rows.
For rare outcomes, a substantially larger total sample may be needed to obtain enough events for informative calibration and discrimination assessments.
The distinction between outcome events and total sample size is therefore central to external validation planning.
Is the Rule of 100 Events and 100 Nonevents Sufficient?
A recommendation of at least 100 events and 100 nonevents has frequently appeared in discussions of external validation for binary prediction models.
It can be useful as a preliminary indication that very small validation datasets may produce unstable performance estimates.
However, it does not guarantee sufficiently precise calibration slopes, C-statistics, or other performance measures.
The required sample can be larger or smaller depending on anticipated performance, the distribution of predicted risks, and the target confidence-interval widths.
Researchers should therefore treat the recommendation as a heuristic rather than a universally sufficient minimum.
How Can You Calculate Sample Size From a Desired Confidence Interval Width?
A precision-based validation calculation begins by identifying a performance parameter and the maximum uncertainty that would remain acceptable.
For a parameter whose estimator has an approximately normal sampling distribution, a general planning relationship is:
The variance of a C-statistic differs from that of a calibration intercept or slope. Consequently, each performance measure requires an appropriate planning calculation.
Why Can the Same Number of Events Produce Different Precision?
Event count is important, but it does not capture the entire distribution of predicted probabilities.
Consider two validation datasets containing the same number of events and nonevents.
In one dataset, the model produces predicted risks ranging widely from very low to very high. In the other, nearly every predicted risk is close to the overall event rate.
The datasets may provide different information for estimating calibration slope and discrimination.
Consequently, sample-size planning may require assumptions about the distribution of predicted risks or the model's linear predictor, in addition to outcome prevalence.
How Does the Intended Validation Population Matter?
External validation should evaluate performance in a population relevant to the model's intended use.
A model developed among students at one university may be evaluated at another institution with different admission policies, academic programs, or support systems.
Such differences may influence outcome prevalence, predictor distributions, and the relationship between predictors and outcomes.
These changes can affect calibration and discrimination independently of sample size.
A large validation sample drawn from an inappropriate population does not necessarily provide useful evidence about performance in the intended deployment setting.
What Is the Difference Between Temporal and Geographic External Validation?
Temporal validation evaluates a model using observations from a later period than those used for development.
Geographic validation evaluates the model in a different location or institutional setting.
Both can provide useful evidence about performance beyond the original development data, although they address different potential sources of variation.
For example, a dropout model developed using one academic year's records may be temporally validated using a later cohort from the same university.
Geographic validation might instead evaluate the model at a different university.
The sample-size requirement depends on the performance measures and precision targets in the selected validation population, not simply on the label assigned to the validation design.
Should the Original Prediction Model Be Recalibrated Before Validation?
For a clean assessment of external performance, the original model should first be applied as specified, without using the validation outcomes to modify its coefficients.
This allows researchers to estimate how the existing model performs in the new population.
If important calibration problems are identified, recalibration or model updating may subsequently be considered.
However, updating changes the statistical task. Estimating new parameters requires additional information and appropriate validation of the updated model.
Researchers should distinguish the initial external validation from subsequent model updating and evaluation.
How Does External Validation Differ From Internal Validation?
Internal validation evaluates the modeling process using information derived from the development dataset.
Bootstrap resampling and cross-validation are common internal validation methods.
These procedures can estimate optimism and assess the stability of model development, but they do not establish performance in a genuinely independent population.
External validation uses independent observations not involved in fitting, selecting, or tuning the original model.
The sample size for prediction model development is therefore a different planning question from the sample size required to estimate external performance precisely.
Should Validation Data Be Used for Hyperparameter Tuning?
Using external validation outcomes to tune model parameters, select predictors, or repeatedly choose among candidate models compromises the independence of the evaluation.
Performance estimates may become optimistic because the validation dataset has influenced model development.
If researchers use the validation data to update the model, they should report that process transparently and obtain an appropriate independent evaluation of the updated model where feasible.
External validation should not become an unacknowledged extension of training.
How Should Sample Size Be Planned for Continuous-Outcome Models?
For continuous outcomes, validation may focus on prediction error, calibration, explained variation, and other performance measures appropriate to numerical predictions.
For example, a model predicting examination scores might be assessed using mean squared prediction error, calibration intercept and slope, and explained variance.
There is no event count for ordinary continuous outcomes.
Sample-size planning should instead reflect the variability of the relevant performance estimators and the precision required for their interpretation.
Methods for binary-outcome validation should not be transferred automatically to continuous-outcome models.
What About Time-to-Event Prediction Models?
Time-to-event prediction models may require validation of survival probabilities at specified horizons, calibration, discrimination, and prediction error.
Observed event counts, censoring, follow-up duration, and the distribution of predicted risks affect the precision of these estimates.
For example, validating a model predicting withdrawal within two years requires adequate outcome follow-up and appropriate handling of students whose event status is not fully observed.
The sample-size method should correspond to the selected survival performance measures and censoring assumptions.
What If the Validation Dataset Is Clustered?
External validation may involve observations from multiple schools, hospitals, clinics, or other institutions.
Participants within the same institution may share characteristics that affect outcomes and model performance.
Ignoring clustering can lead to misleading uncertainty estimates.
When the objective is to assess performance across settings, the number and diversity of clusters may be important in addition to the total number of individuals.
Researchers may need multilevel or internal-external validation approaches, with sample-size planning that reflects between-setting heterogeneity.
Can Simulation Be Used to Plan External Validation Sample Size?
Yes. Simulation can be particularly useful when analytical formulas do not adequately represent the desired validation measures.
Researchers can specify plausible distributions of predicted risks and outcomes, generate validation datasets of different sizes, and calculate performance measures repeatedly.
The resulting distributions can be used to assess expected confidence-interval widths, coverage, and the probability of achieving a specified precision target.
Simulation assumptions should be informed by relevant prior data and examined through sensitivity analysis.
A simulation is not inherently more credible than an analytical calculation when its assumed data-generating process is poorly justified.
What If the Available Validation Dataset Is Too Small?
Researchers should first identify which performance measures can be estimated with useful precision.
A small dataset may permit a preliminary estimate of discrimination but provide insufficient information for detailed calibration assessment, particularly when the outcome is rare.
Possible responses include collecting additional validation observations, extending follow-up, or combining compatible independent datasets with appropriate attention to heterogeneity.
If the available sample remains limited, researchers should report confidence intervals and acknowledge the uncertainty rather than claim that the model has been definitively validated.
A nonsignificant difference between observed and predicted outcomes is not proof of adequate calibration.
How Should Uncertainty in Validation Assumptions Be Handled?
Before validation, researchers may have incomplete information about outcome prevalence, anticipated C-statistic, calibration slope, or the distribution of predicted risks.
These uncertainties can materially affect the required sample.
A sample-size sensitivity analysis can examine how the requirements change under credible alternatives.
The final sample-size justification should identify the primary performance targets, the assumptions supporting them, and the consequences of uncertainty.
