03 · What You Need to Know
How Sample Size Determines the Complexity a Prediction Model Can Support
What Is a Candidate Predictor?
A candidate predictor is a characteristic considered for inclusion during model development because it may contain information about the outcome being predicted.
Examples include age, baseline disease severity, laboratory results, previous academic performance, enrollment characteristics, and prior service utilization.
Candidate predictors should be identified according to the prediction task, prior evidence, measurement quality, and availability at the intended decision point.
They are not necessarily identical to the predictors retained in the final model.
Researchers may begin with a prespecified candidate set and subsequently estimate, shrink, or select coefficients through an appropriate modeling procedure.
The distinction matters because the full development process consumes information, including when some candidate predictors are ultimately excluded.
Why Is the Number of Participants Not Enough?
Sample size adequacy depends on how much information the observations provide about the outcome and predictor relationships.
For binary outcomes, both event and non-event observations matter. A model developed with 500 events and 500 non-events has a different information structure from one with 20 events and 980 non-events.
For time-to-event outcomes, event timing, censoring, and the prediction horizon also influence the available information.
For continuous outcomes, residual variation and anticipated explained variance are important considerations.
Consequently, the same participant count may support different model complexities depending on the outcome structure.
Researchers should not decide how many predictors to include solely by dividing the number of participants by an arbitrary constant.
What Is the Difference Between a Predictor and a Predictor Parameter?
A predictor is a measured characteristic. A predictor parameter is a coefficient or other estimated quantity used to represent its relationship with the outcome.
A continuous predictor entered linearly into a regression model ordinarily contributes one coefficient.
A categorical predictor with four categories ordinarily contributes three coefficients when represented using reference-category indicator coding.
Nonlinear terms, interactions, and flexible transformations can require additional parameters.
For example, a model containing ten named predictors may require 18 or more predictor coefficients, depending on how those predictors are represented.
Sample size planning should therefore count candidate predictor parameters rather than relying only on variable names.
How Does Predictor Representation Affect Parameter Count?
| Predictor Representation |
Illustrative Parameter Count |
Important Consideration |
| One continuous predictor entered linearly |
1 |
Assumes a linear predictor relationship on the model scale |
| One binary predictor |
1 |
Requires sufficient observations in relevant categories |
| One categorical predictor with four levels |
3 |
Reference-category coding uses three coefficients |
| Continuous predictor modeled using a four-parameter spline basis |
4 |
Flexible functional forms increase complexity |
| Interaction between two linear predictors |
1 additional parameter |
Main effects are generally also included |
| Three separate binary predictors |
3 |
Correlation and distribution still influence estimation |
These counts are illustrative and depend on the specific parameterization. Penalized and machine-learning models may require additional considerations beyond simply counting conventional regression coefficients.
What Is the Events-per-Variable Rule?
The events-per-variable rule, often abbreviated EPV, is a historical heuristic used to assess whether a regression model has sufficient outcome information relative to its complexity.
A frequently cited recommendation is at least ten events per predictor variable or parameter.
However, this recommendation arose from particular simulation settings and should not be interpreted as a universal threshold for model adequacy.
Later methodological research demonstrated that performance depends on the number of candidate parameters, predictor distributions, anticipated model fit, and other characteristics.
Some models may be adequately estimated with fewer than ten events per parameter, while others may require substantially more.
The ratio can describe a dataset, but it cannot replace a tailored sample size calculation.
Why Is Counting Only Final Predictors Misleading?
Suppose researchers examine 40 candidate predictors and use stepwise regression to retain six statistically significant variables.
Calculating EPV using only the six retained predictors ignores the information used to screen and select among the original 40.
The selection process can exploit random associations, producing unstable coefficients and optimistic apparent performance.
Removing variables after examining the outcomes does not restore the information that would have been available if only six predictors had genuinely been prespecified.
For sample size planning, researchers should consider the original candidate parameter set and the complete selection procedure.
The importance of selecting candidate predictors before data collection follows partly from this problem.
What Is Overfitting?
Overfitting occurs when a model captures sample-specific patterns that do not generalize adequately to new observations.
When model complexity is high relative to available information, estimated coefficients may become unstable and predicted risks may be overly extreme.
A model may consequently achieve excellent apparent performance in its development dataset while performing less well elsewhere.
Overfitting is not restricted to highly complex machine-learning algorithms. Conventional logistic and Cox regression can also overfit.
The risk increases when numerous predictors, transformations, interactions, or data-driven selection decisions are considered without sufficient information.
What Is Global Shrinkage?
Global shrinkage reduces the magnitude of estimated predictor coefficients to limit overfitting and overly extreme predictions.
In regression-based prediction models, shrinkage factors between zero and one may be applied to estimated coefficients, with the intercept or baseline risk adjusted appropriately afterward.
A factor close to one indicates relatively little shrinkage, whereas a smaller factor indicates stronger correction.
Modern sample size approaches may specify a target anticipated global shrinkage factor, commonly at least 0.90 in particular development frameworks.
However, 0.90 is a planning criterion within those frameworks, not a universal guarantee of adequate model performance.
The appropriate criterion depends on the outcome, modeling strategy, and intended predictions.
What Do Modern Sample Size Methods Consider?
Modern approaches to prediction-model sample size planning aim to determine whether the available information can support sufficiently stable model estimation.
For binary and time-to-event regression models, methods proposed by Riley and colleagues consider the number of candidate predictor parameters, anticipated model fit, overall outcome risk, and criteria for limiting overfitting.
Relevant criteria include targeting an acceptable degree of shrinkage, limiting optimism in model fit, and estimating overall outcome risk with adequate precision.
The required sample size is determined by the most demanding applicable criterion.
These methods require assumptions based on prior evidence or defensible planning estimates.
They are more informative than selecting a sample size solely because it satisfies a fixed EPV threshold.
How Does Anticipated Model Performance Affect Sample Size?
Sample size requirements depend partly on how strongly the candidate predictors are expected to predict the outcome.
When anticipated predictive information is weak, a model with many parameters may be particularly susceptible to overfitting.
When prior evidence supports stronger predictive relationships, the same parameter count may require a different sample size under a specified planning framework.
However, researchers should not assume unrealistically strong performance merely to justify a convenient sample size.
Anticipated performance should be based on relevant previous studies, existing models, or cautiously justified estimates.
Sensitivity calculations can examine how requirements change under less favorable assumptions.
How Does Outcome Frequency Affect Predictor Capacity?
For binary outcomes, rare events can limit the information available to estimate predictor-outcome relationships.
For example, a cohort of 2,000 participants with an event frequency of 2% would be expected to produce approximately 40 events.
A model containing 30 candidate predictor parameters would require careful evaluation because the event information may be inadequate for stable estimation.
However, event count alone does not determine adequacy. Non-event information, predictor distributions, and anticipated model fit also matter.
Researchers should consider the full information structure rather than assuming that adding non-events indefinitely compensates for very few events.
How Does Censoring Affect Time-to-Event Models?
Time-to-event models require information about when events occur and how long participants remain under observation.
Administrative censoring and loss to follow-up can reduce the event information available at the intended prediction horizon.
A dataset spanning several years may still provide limited information about long-term prognosis if most participants have short follow-up.
Researchers should estimate expected event accrual using realistic recruitment, follow-up, and censoring assumptions.
When competing events are relevant, the intended estimand and competing-event probabilities should also be considered.
The principles governing outcome-event requirements in prognostic studies apply directly to model development.
Can Penalized Regression Support More Predictors?
Penalized regression can reduce coefficient instability and overfitting by constraining parameter estimates.
Ridge regression shrinks coefficients toward zero, while lasso may shrink some coefficients exactly to zero. Elastic net combines aspects of these approaches.
These methods may be particularly useful when candidate predictors are correlated or numerous.
However, their use does not mean that an arbitrarily large predictor set is supported by a small dataset.
Penalty selection itself requires appropriate tuning and validation.
Researchers should distinguish reducing model variance through regularization from creating additional information about the outcome.
Does Machine Learning Remove the Need for Sample Size Planning?
No. Machine-learning methods also depend on sufficient representative data.
Flexible algorithms may require substantial information to estimate complex relationships and avoid unstable predictions.
However, traditional regression parameter counts do not necessarily describe the effective complexity of every machine-learning algorithm.
Sample size planning may therefore require learning-curve analyses, simulation, prior performance evidence, or other methods appropriate to the algorithm and prediction task.
Researchers should avoid applying regression-based EPV calculations mechanically to every machine-learning model.
Regardless of the algorithm, independent performance evaluation remains essential.
Should Continuous Predictors Be Categorized to Save Parameters?
Arbitrarily categorizing continuous predictors is generally not a reliable strategy for reducing model complexity.
For example, converting age into younger and older categories may discard information and introduce dependence on an arbitrary threshold.
Although categorization may reduce the number of estimated coefficients in some specifications, it can also weaken predictive performance and obscure nonlinear relationships.
Researchers should consider scientifically justified functional forms and the amount of data available to estimate them.
When nonlinear modeling is appropriate, the additional parameters should be incorporated into sample size planning.
What If Predictors Are Highly Correlated?
Correlated predictors may provide overlapping information, making individual coefficient estimates unstable in some models.
However, correlation does not automatically make the predictors unsuitable for prediction.
Penalization, carefully justified predictor reduction, or alternative representations may help manage redundancy.
Researchers should consider whether the predictors add useful information and whether their measurement is practical.
Removing variables solely because a correlation exceeds an arbitrary threshold may discard relevant predictive information.
How Should Interactions and Nonlinear Terms Be Planned?
Interactions and nonlinear functions increase flexibility but also increase the information required for stable estimation.
For example, including an interaction between treatment status and baseline severity adds parameters beyond their main effects.
Restricted cubic splines may represent nonlinear relationships more flexibly than a single linear coefficient, but they also require additional degrees of freedom.
Researchers should prioritize scientifically plausible terms and avoid unrestricted searches for interactions when outcome information is limited.
The complete candidate model specification should be considered during sample size planning, not reconstructed after favorable results are observed.
Can Internal Validation Compensate for Insufficient Data?
Internal validation can estimate optimism and identify unstable performance, but it does not eliminate inadequate sample size.
Bootstrapping and cross-validation reuse information from the available development dataset rather than generating new independent outcome observations.
A model developed with too little information may remain unstable despite technically correct validation.
Researchers should therefore evaluate sample size adequacy before development and use internal validation to assess the resulting modeling procedure.
The distinction between internal validation and train-test splitting is relevant to making efficient use of limited observations.
What Should Researchers Do When the Dataset Cannot Support the Proposed Predictors?
Possible responses include collecting additional data, reducing unnecessary model complexity, using a smaller scientifically justified candidate set, or validating an existing model instead of developing a new one.
Researchers may also consider combining compatible datasets or selecting a narrower prediction task.
However, reducing predictors through outcome-driven significance screening can introduce selection bias and does not automatically solve the information problem.
Any simplification should be justified before examining the final model's apparent performance whenever feasible.
When the available information remains insufficient, exploratory results should be presented with appropriate uncertainty rather than described as a reliable deployable model.
How Should Predictor Capacity Be Reported?
Researchers should report the planned and observed sample sizes, event frequencies, and candidate predictor parameter counts.
The sample size method should be identified, including assumptions about anticipated model performance and overfitting.
For time-to-event outcomes, relevant follow-up and censoring assumptions should be documented.
Any changes to candidate predictors or functional forms should be distinguished from the prespecified plan.
Reporting should explain why the available information was considered sufficient for the proposed development strategy rather than relying solely on a conventional EPV calculation.
