03 · What You Need to Know
How to Plan Candidate Predictors Without Unnecessarily Restricting Model Development
What Is a Candidate Predictor?
A candidate predictor is a characteristic considered for inclusion in a prediction model because it may contain information about the target outcome.
Examples include age, previous diagnoses, laboratory measurements, prior academic performance, enrollment characteristics, and other information relevant to the prediction task.
Candidate predictors are not necessarily the same as predictors retained in the final model.
Researchers may begin with a justified set of candidate variables and subsequently use an appropriate modeling strategy to estimate or regularize their contributions.
The distinction matters because sample size planning should account for the information used throughout development, including candidate predictor parameters and any selection procedures.
Why Should Candidate Predictors Be Considered Before Data Collection?
Early planning helps ensure that potentially useful information is measured consistently and at the correct time.
For example, researchers developing a model to predict postoperative complications may need particular pretreatment measurements. If these are not collected before surgery, they may be impossible to reconstruct reliably afterward.
Similarly, a model intended to estimate student withdrawal risk at enrollment requires predictors available at enrollment rather than measurements collected later.
Prespecification also allows researchers to estimate model complexity and determine whether the anticipated sample size can support the proposed development strategy.
Without early planning, researchers may discover that important predictors are unavailable or that the collected dataset contains too little outcome information for the intended model.
Does Prespecification Mean Choosing the Final Model in Advance?
No. Prespecifying candidate predictors does not necessarily require fixing every coefficient, transformation, interaction, or final inclusion decision before observing the data.
Researchers should distinguish the scientific selection of candidate information from the statistical estimation and modeling procedures used during development.
For example, investigators may prespecify age, baseline severity, and previous treatment history as candidate predictors while allowing a penalized regression method to estimate their contributions.
They may also prespecify a strategy for examining nonlinear relationships or selecting tuning parameters.
The important requirement is that the modeling strategy be scientifically defensible and evaluated in a way that accounts for its flexibility.
How Should Candidate Predictors Be Identified?
Researchers should begin with the intended prediction task and relevant existing evidence.
Previous studies, systematic reviews, established models, and substantive expertise can identify characteristics plausibly associated with the outcome.
Practical considerations also matter. Predictors should be measurable with sufficient reliability and available when predictions are intended to be made.
For example, a highly informative laboratory test may be unsuitable for a model intended for settings where the test is rarely available.
Researchers should document why each candidate predictor is considered relevant rather than assuming that every variable in a database deserves inclusion.
What Criteria Should Guide Candidate Predictor Selection?
| Criterion |
Question to Ask |
Why It Matters |
| Scientific relevance |
Is there a plausible or previously documented relationship with the outcome? |
Supports a defensible candidate set |
| Prediction-time availability |
Will the information exist when the prediction is needed? |
Prevents future-information leakage |
| Measurement reliability |
Can the predictor be measured consistently? |
Reduces measurement-related instability |
| Practical feasibility |
Is the predictor accessible at acceptable cost and burden? |
Supports eventual implementation |
| Incremental information |
Could the predictor contribute information beyond existing predictors? |
Helps justify additional complexity |
| Model complexity |
How many parameters might its representation require? |
Supports sample size planning |
| Applicability |
Will its meaning and measurement remain suitable in the target population? |
Supports transportability |
These considerations should be evaluated together. A predictor can be statistically informative but impractical, while an easily measured characteristic may contribute little additional predictive information.
Why Is Predictor Availability at the Decision Point Essential?
A prediction model should use information that would actually be available when the prediction is generated.
For example, a model intended to estimate hospital readmission risk at discharge may use information collected during the hospitalization.
However, it should not use information obtained after discharge unless the intended prediction time is explicitly moved to a later point.
Similarly, a model predicting first-year academic withdrawal at enrollment cannot legitimately use final examination scores obtained months later.
Including future information can produce apparently impressive performance that cannot be reproduced in practice.
The timing of candidate predictors should therefore be aligned with the defined starting point for prognosis.
What Is Data-Driven Predictor Selection?
Data-driven selection uses observed data to determine which variables or features enter a model.
Methods include stepwise selection, univariable screening, penalized regression, recursive feature elimination, and algorithm-specific feature selection procedures.
These approaches differ substantially in their assumptions and statistical properties.
Some may be useful for managing complex candidate predictor sets, particularly when combined with appropriate regularization and validation.
However, selection procedures can exploit chance associations in development data and produce unstable models.
The presence of a formal algorithm does not eliminate the need to evaluate selection-related optimism.
Why Is Univariable Screening Often Discouraged?
Univariable screening examines each candidate predictor separately and retains variables meeting a specified association threshold.
For example, researchers may include only predictors with p-values below 0.05 in individual logistic regressions.
This strategy can be problematic because a predictor's contribution to a multivariable model may differ from its univariable association.
A predictor with weak marginal association may provide useful information conditional on other variables.
Conversely, a strongly associated predictor may contribute little incremental information beyond characteristics already included.
Univariable screening also introduces selection uncertainty that is often ignored in subsequent performance evaluation.
Researchers should not treat statistical significance as a universal criterion for predictive usefulness.
Why Can Stepwise Selection Produce Unstable Models?
Stepwise procedures repeatedly add or remove variables according to statistical criteria.
The selected model may depend strongly on random variation in the development sample, especially when candidate predictors are correlated or outcome events are limited.
Repeated selection can also produce overly optimistic coefficient estimates and performance when the selection process is ignored.
Although stepwise procedures remain common, their convenience should not be confused with methodological superiority.
Prespecified modeling strategies, penalization, and appropriate validation may offer more defensible alternatives in many prediction settings.
The best approach depends on the number and structure of candidate predictors and the available information.
Can Penalized Regression Help With Predictor Selection?
Penalized regression methods constrain coefficient estimation to reduce overfitting.
Ridge regression shrinks coefficients toward zero without ordinarily setting them exactly to zero.
Lasso regression can shrink some coefficients to zero, effectively performing variable selection under its penalty structure.
Elastic net combines aspects of ridge and lasso penalties.
These approaches can be useful when candidate predictors are correlated or the model contains numerous parameters.
However, their performance depends on appropriate tuning, data structure, and sample size.
Penalization does not remove the need for adequate outcome information or independent validation.
Should Highly Correlated Predictors Be Removed Automatically?
Not necessarily. Correlation among predictors can affect coefficient stability and interpretation, but its consequences depend on the modeling objective.
In prediction research, correlated predictors may contain overlapping information, and including all of them may increase complexity without substantial improvement.
However, automatically removing one variable from every correlated pair can discard useful information or produce arbitrary decisions.
Researchers should consider measurement reliability, practical availability, predictive contribution, and the modeling method.
Penalization or carefully justified predictor reduction may be appropriate.
Correlation thresholds should not be treated as universal rules for inclusion or exclusion.
How Should Continuous Predictors Be Represented?
Continuous predictors often contain more information than arbitrary categories created from them.
For example, converting age or a laboratory value into a binary high-versus-low category can discard information and introduce sensitivity to the chosen threshold.
Researchers should generally consider retaining continuous measurements and modeling their relationships appropriately.
When nonlinear associations are plausible, restricted cubic splines or other flexible functions may be considered.
However, flexible representations require additional parameters and sufficient data.
The proposed functional form should be considered during candidate predictor planning because it affects model complexity.
Should Interactions Be Specified in Advance?
Interactions may be scientifically relevant when the predictive contribution of one characteristic depends on another.
For example, the relationship between a biomarker and an outcome may vary according to treatment status or disease stage.
However, interactions increase model complexity and can be difficult to estimate reliably when outcome information is limited.
Researchers should prioritize interactions supported by substantive knowledge or a clearly justified modeling strategy.
Searching extensively for interactions after observing outcomes can increase overfitting and requires appropriate validation.
Exploratory interaction discovery should be identified as such.
How Many Candidate Predictors Can the Dataset Support?
The answer depends on sample size, outcome frequency, anticipated predictive performance, predictor distributions, and the number of parameters required.
A model with ten named predictors may require considerably more than ten parameters if categorical variables, nonlinear terms, and interactions are included.
Traditional rules such as ten events per variable are not sufficient as universal sample size justifications.
Modern prediction-model sample size approaches consider overfitting, anticipated model fit, and precision of predicted risks.
Researchers should determine how many candidate predictors the available data can support before committing to a complex development strategy.
What If the Dataset Already Exists?
Researchers using existing records cannot always control which variables were collected.
Nevertheless, they can prespecify a scientifically justified candidate predictor set before examining predictor-outcome associations.
They should review data dictionaries, measurement timing, missingness, and the intended prediction application.
Available variables should not automatically be included merely because they appear in the database.
Likewise, researchers should not select the candidate set solely by testing every available variable against the outcome.
When exploratory selection is necessary, its scope and validation strategy should be documented transparently.
Can Machine Learning Select Predictors Automatically?
Some machine-learning algorithms can identify patterns or assign different importance to candidate features during training.
However, automatic selection does not eliminate the need for substantive judgment about predictor validity, timing, availability, and ethical implications.
A model may identify a variable that strongly predicts the outcome because it indirectly encodes information obtained after the intended prediction time.
It may also rely on institutional practices or proxy characteristics that do not generalize well.
Researchers should evaluate the candidate feature set before modeling and assess the entire algorithmic development process during validation.
Feature importance measures should not automatically be interpreted as evidence of causal influence.
Why Must Predictor Selection Be Included in Internal Validation?
Predictor selection is part of model development whenever it uses information from observed outcomes.
If researchers select predictors using the entire dataset and then perform cross-validation on the selected set, the evaluation folds have already influenced development.
This introduces information leakage and can produce optimistic performance estimates.
Outcome-informed feature selection should therefore be repeated within the training portion of each resampling iteration.
Similarly, hyperparameter tuning should be appropriately separated from final internal performance evaluation.
The principles discussed in internal validation and train-test splitting apply to the complete predictor-selection pipeline.
Should Predictor Selection Be Repeated During External Validation?
Not when the objective is to evaluate the original fixed model.
External validation should apply the existing prediction function without using validation outcomes to choose new predictors or modify its parameters.
If researchers change the predictor set after examining validation results, they are updating or developing a modified model.
The original model's validation performance should be reported separately.
The revised model then requires appropriate evaluation.
This distinction preserves the meaning of external validation of prediction models.
What If an Important Predictor Is Too Expensive to Measure?
Predictor selection should consider practical implementation.
A variable may improve statistical performance but require specialized equipment, invasive procedures, substantial staff time, or costly data collection.
Researchers should evaluate whether its incremental predictive contribution justifies these requirements.
For example, a modest improvement in discrimination may not justify an expensive laboratory test when an inexpensive model already provides adequate predictions for the intended decision.
However, a costly predictor may be worthwhile when it meaningfully improves decisions in a high-risk population.
The trade-off between predictor availability and measurement cost should be considered during development rather than only after publication.
How Can Predictor Selection Affect Fairness?
Predictors may encode socioeconomic circumstances, institutional practices, or historical inequalities.
For example, a variable reflecting previous access to services may predict an outcome partly because access itself has been unequal.
Removing a sensitive variable does not necessarily remove its influence because other predictors may act as proxies.
Conversely, excluding relevant information can sometimes reduce performance or conceal differences in risk across groups.
Researchers should examine measurement validity, subgroup performance, and the consequences of using predictions in decisions.
Fairness cannot be established solely through a particular variable inclusion or exclusion rule.
How Should Candidate Predictor Selection Be Documented?
The protocol should identify the intended prediction task and explain the rationale for the candidate predictor set.
Researchers should document predictor definitions, measurement windows, coding, anticipated transformations, and practical availability.
They should also specify the planned modeling strategy, including whether selection or penalization will be used.
Changes made after examining the data should be distinguished from prespecified decisions.
Reporting should identify the original candidate predictors, the final model specification, and the procedures used to evaluate selection-related optimism.
TRIPOD+AI provides relevant reporting guidance for clinical prediction-model research.
