03 · What You Need to Know
Determining the Sample Size Needed to Develop a Reliable Prediction Model
What Is Prediction Model Development?
Prediction model development involves using observed data to estimate a mathematical relationship between predictors and an outcome so that predictions can be generated for individuals whose outcomes are not yet known.
For example, a university might develop a model that estimates the probability of student dropout from prior academic performance, attendance, enrollment characteristics, and engagement measures.
The intended output is an individual prediction, such as a probability of dropout, rather than necessarily a causal explanation of why dropout occurs.
This distinction matters because a predictor can improve prediction without being a causal determinant of the outcome. Conversely, a scientifically important explanatory variable may contribute little additional predictive information after other variables are included.
The primary sample-size objective is therefore to support a model that performs adequately when applied beyond the development observations.
Why Is Sample Size Important for Prediction Models?
Prediction models estimate relationships from a finite dataset. When the amount of information is limited relative to model complexity, the fitted model may capture random patterns specific to the development sample.
This phenomenon is known as overfitting.
An overfitted model may perform impressively when evaluated using the same data on which it was developed, but less favorably when applied to new individuals.
Possible consequences include unstable predictor coefficients, excessively extreme predicted probabilities, poor calibration, and optimistic estimates of predictive performance.
Increasing sample size generally provides more information, but the amount needed depends on the prediction task and modeling procedure.
What Determines the Required Development Sample?
Several factors influence the information required for prediction model development.
| Planning factor |
Why it matters |
| Outcome type |
Continuous, binary, count, and time-to-event outcomes require different estimation and planning methods. |
| Outcome frequency |
Rare binary events may provide limited information even when total sample size is large. |
| Candidate predictor parameters |
Additional coefficients and model flexibility increase complexity. |
| Anticipated model performance |
The expected strength of predictive relationships affects overfitting and sample-size criteria. |
| Acceptable shrinkage |
A stricter requirement for limiting coefficient overfitting may increase the required sample. |
| Predictor distributions |
Rare categories, restricted variation, and correlated predictors can reduce usable information. |
| Modeling procedure |
Variable selection, nonlinear functions, interactions, and tuning can increase effective complexity. |
These factors should be evaluated jointly. A large sample does not necessarily compensate for extremely rare outcomes or poorly represented predictor categories.
Why Is the Number of Candidate Predictor Parameters Important?
Every candidate parameter introduces flexibility into a model. Greater flexibility may improve the representation of genuine relationships, but it also increases the opportunity to fit sampling noise.
Importantly, a predictor variable does not always correspond to one parameter.
A categorical predictor with five levels may require four coefficients. A continuous predictor represented using nonlinear spline terms may require several coefficients. Interactions introduce further parameters.
For example, a model with eight named predictors could contain fifteen or more predictor parameters depending on their specification.
The number of candidate predictor parameters should therefore be established before calculating the required sample.
For data-driven predictor selection, the original candidate set matters because the selection process itself can contribute to overfitting.
Why Does Outcome Frequency Matter for Binary Prediction?
Binary prediction models estimate the probability of an event, such as dropout or treatment failure.
When the outcome is uncommon, relatively few observations may contain the event of interest.
For example, a dataset containing 2,000 students with a dropout prevalence of 2% would contain approximately 40 dropout events on average.
That may provide limited information for a complex prediction model, particularly if the candidate predictors include rare categories or interactions.
The number of outcome events needed for logistic regression cannot be determined using a universal minimum.
Both events and nonevents contribute information, and their distribution across predictor values is consequential.
Is Ten Events per Variable Sufficient for Prediction Model Development?
The traditional ten-events-per-variable rule is a historical heuristic rather than a universally valid sample-size requirement.
It does not directly account for anticipated model performance, the acceptable degree of overfitting, or the precision of estimated outcome risk.
Van Smeden and colleagues (2019) investigated the limitations of events-per-variable criteria for binary logistic prediction models.
Their work supports moving beyond fixed event-to-parameter ratios toward criteria that more directly address prediction model performance.
Consequently, the ten events per variable rule should not be the sole justification for a development sample.
What Is the Riley Sample-Size Approach?
Riley and colleagues (2019) proposed minimum sample-size criteria for developing multivariable prediction models with continuous, binary, and time-to-event outcomes.
Their approach considers multiple criteria rather than selecting a sample solely from a fixed number of observations or events per predictor.
For binary-outcome prediction models, important criteria include limiting overfitting through a target global shrinkage factor, restricting optimism in model fit, and estimating the overall outcome risk with sufficient precision.
The final recommended sample is generally the largest requirement arising from the applicable criteria.
These calculations require assumptions about anticipated model performance and candidate predictor complexity. They are not free of uncertainty simply because the resulting formula is more sophisticated.
What Is a Global Shrinkage Factor?
Shrinkage describes the reduction of estimated regression coefficients toward zero to address overfitting.
In conventional prediction modeling, a global shrinkage factor below one indicates that the fitted coefficients may need to be reduced to improve their expected performance in new data.
A target shrinkage factor of at least 0.90 is commonly used in the sample-size framework proposed by Riley and colleagues, although the appropriate criterion depends on the planning objective.
A factor of 0.90 should not be interpreted as 90% predictive accuracy.
Rather, it is a criterion related to the anticipated extent of coefficient overfitting.
How Can Shrinkage Be Used in a Sample-Size Calculation?
For a conventional maximum-likelihood logistic prediction model, one approximation used in shrinkage-based planning relates sample size to the number of candidate predictor parameters and anticipated Cox-Snell R-squared.
The expression should not be applied indiscriminately to machine-learning algorithms, penalized procedures, or other outcome types. Its assumptions and interpretation belong to the specified conventional prediction-model framework.
Why Is Anticipated R-Squared Needed?
Anticipated model performance provides information about how much of the outcome variation the candidate predictors are expected to explain.
For binary outcomes, Cox-Snell R-squared is one measure of overall model fit that can inform sample-size calculations.
Its possible maximum depends on the outcome distribution, so it is not interchangeable with every other pseudo-R-squared measure.
Researchers may obtain plausible performance estimates from previous prediction studies, systematic reviews, or comparable datasets.
However, apparent performance from a small development study can be optimistic. Borrowed estimates should be evaluated for transportability to the proposed population.
If anticipated performance is uncertain, several plausible values should be examined through sensitivity analysis.
Why Does Estimating Overall Outcome Risk Matter?
A prediction model generally includes an intercept or baseline component that influences the overall level of predicted outcomes.
For binary prediction, the overall event proportion provides information relevant to this component.
If the event rate is estimated imprecisely, the model's overall predicted risk may also be uncertain.
One planning criterion therefore concerns the precision with which the overall outcome risk can be estimated.
For a simple independent binary-outcome setting, an approximate calculation is:
Why Must Several Sample-Size Criteria Be Considered?
A development sample might satisfy one criterion while failing another.
For example, a sample may be large enough to estimate overall event prevalence reasonably precisely but too small to limit coefficient overfitting.
Conversely, a relatively simple model with strong anticipated performance may satisfy a shrinkage criterion while still producing an imprecise estimate of a rare outcome's overall frequency.
Modern planning therefore considers multiple requirements and identifies the most demanding applicable criterion.
The numerical result should then be evaluated against the assumptions supporting each calculation.
How Does Sample-Size Planning Differ for Continuous Outcomes?
Continuous-outcome prediction models estimate numerical quantities such as examination scores, blood pressure, or completion time.
Unlike binary-outcome models, they do not have a natural count of outcome events.
The relevant information depends on residual variation, anticipated explained variance, candidate parameter complexity, and the precision of important model quantities.
Riley and colleagues (2019) provide criteria for continuous-outcome prediction models that consider overfitting and estimation precision.
A fixed number of observations per predictor is not a substitute for these calculations.
What About Time-to-Event Prediction Models?
Time-to-event models predict whether or when an event occurs, accounting for follow-up time and censoring.
Examples include predicting time until student withdrawal, disease recurrence, or equipment failure.
Sample-size planning must consider the number of observed events, the duration of follow-up, censoring, candidate parameters, and anticipated model performance.
Two studies with the same total sample size can provide different information when their event frequencies and follow-up patterns differ.
Methods developed specifically for time-to-event prediction should therefore be used.
Can Machine-Learning Models Use the Same Sample-Size Formulas?
Not necessarily. Random forests, gradient boosting, neural networks, and other machine-learning methods can have effective complexity that is not adequately described by a conventional regression coefficient count.
The required sample may depend on the algorithm, tuning strategy, predictor dimensionality, signal strength, class imbalance, and intended performance measure.
For these methods, simulation or learning-curve analysis may help evaluate how model performance changes with sample size.
However, learning curves estimated from an existing small dataset cannot reliably establish performance at arbitrarily larger sample sizes without additional assumptions.
Researchers should avoid assuming that an algorithm capable of fitting high-dimensional data can necessarily generalize from a small development sample.
Does Regularization Reduce the Required Sample Size?
Regularization methods such as ridge regression, lasso, and elastic net can reduce some consequences of overfitting.
They may improve predictive performance when model complexity is substantial relative to the available information.
However, regularization does not guarantee adequate calibration or reliable performance in new populations.
Hyperparameter tuning also uses information from the development data and should be incorporated into validation.
Sample-size planning should reflect the complete modeling procedure rather than assume that penalization eliminates information requirements.
Should the Development Sample Be Split Into Training and Testing Sets?
Randomly splitting a limited development dataset into separate training and testing subsets is not automatically the most efficient approach.
A split reduces the number of observations available for fitting the model, while the test subset may be too small to estimate performance precisely.
Bootstrap validation and appropriate cross-validation can often make more efficient use of limited development data for internal validation.
These procedures should include all relevant model-development steps, including variable selection and hyperparameter tuning, to avoid information leakage.
Internal validation does not replace external validation using an independent dataset.
How Should Missing Data Affect Sample-Size Planning?
Missing predictor and outcome information can reduce the effective information available for development.
Complete-case analysis may discard substantial portions of the sample and can introduce bias when missingness is systematic.
Appropriate imputation methods may retain information under specified assumptions, but they cannot fully recover information absent from the data.
Researchers should anticipate missingness, specify a defensible analysis strategy, and consider its implications for the required recruitment sample.
The development and validation procedures must handle missing data consistently with the intended real-world use of the model.
What If the Available Sample Is Smaller Than Recommended?
A sample-size recommendation may exceed the available data, particularly when outcomes are rare or candidate models are complex.
Researchers can consider collecting additional observations, combining compatible datasets, simplifying the candidate model, or using appropriate penalized methods.
However, reducing predictors should be scientifically justified. Removing potentially useful information solely to meet a numerical rule may undermine the intended model.
If the available sample remains inadequate for the intended objective, the analysis may need to be framed as exploratory or developmental, with explicit limitations concerning uncertainty and generalizability.
Reporting the achieved sample and model performance transparently is preferable to presenting an inadequately supported model as ready for deployment.
How Should Uncertainty in Planning Assumptions Be Addressed?
Sample-size requirements can be sensitive to anticipated outcome prevalence, model performance, candidate parameter count, and acceptable shrinkage.
Researchers should avoid selecting optimistic assumptions merely because they produce a feasible sample.
A sample-size sensitivity analysis can show how requirements change under plausible alternatives.
Where several inputs are uncertain, using evidence-informed ranges provides a more transparent basis for planning than relying on a single unsupported estimate.
