03 · What You Need to Know
How to Recognize a Genuine Prediction Model Research Question
What Is a Prediction Model Study?
A prediction model study investigates a mathematical or algorithmic method for estimating an outcome from information about an individual or other unit of analysis.
The model may use demographic characteristics, clinical measurements, prior performance, behavioral information, or other predictors relevant to the intended application.
For example, a clinical model may estimate the probability of hospitalization within one year. An educational model may estimate the likelihood of academic withdrawal during the first semester.
The defining feature is not the use of a particular statistical algorithm. It is the objective of producing and evaluating predictions for observations beyond those used to develop the model.
Prediction-model research can involve model development, internal validation, external validation, updating, or comparisons between models.
What Types of Outcomes Can Prediction Models Address?
Prediction models can address several outcome structures, provided the model and evaluation methods are appropriate.
| Outcome Type |
Illustrative Prediction Question |
Potential Modeling Approach |
| Binary outcome |
Will the individual experience an event within one year? |
Logistic regression or suitable classification methods |
| Time-to-event outcome |
What is the probability of an event by a specified time? |
Survival prediction models |
| Continuous outcome |
What will the individual's future assessment score be? |
Linear or other regression-based prediction |
| Multicategory outcome |
Which of several outcome categories is most likely? |
Multinomial or other multiclass prediction methods |
| Repeated or dynamic outcome |
How does predicted risk change as new information becomes available? |
Landmark or longitudinal prediction methods |
Researchers should select the outcome structure before choosing an algorithm. A classification method that ignores event timing may be inadequate when participants have different follow-up durations or experience competing events.
How Do Diagnostic and Prognostic Prediction Models Differ?
Diagnostic prediction models estimate the probability that a target condition is currently present.
Prognostic prediction models estimate the probability or timing of an outcome that may occur in the future.
For example, a diagnostic model might combine symptoms and laboratory findings to estimate whether a patient currently has a particular condition.
A prognostic model might use information available at diagnosis to estimate the probability of deterioration within six months.
Both approaches require a clearly defined target population, appropriate predictors, valid outcome or reference classifications, and suitable performance evaluation.
However, prognostic models require particular attention to time zero, follow-up, censoring, and the prediction horizon.
Researchers should not treat a model predicting an existing condition as equivalent to one predicting future disease development.
How Is Prediction Different From Prognostic Factor Research?
Prognostic factor research investigates whether a characteristic is associated with a future outcome.
Prediction-model research evaluates how well information from several characteristics can be combined to estimate outcomes.
For example, a study examining whether prior academic performance is associated with later withdrawal is primarily investigating a prognostic factor.
A study combining prior performance, enrollment characteristics, and other information to estimate individual withdrawal probabilities is investigating prediction.
The distinction concerns the research objective rather than whether regression is used.
A statistically significant association does not establish that a characteristic improves predictive performance.
The difference between prognostic factor and prediction model studies should therefore be established before planning the analysis.
How Is Prediction Different From Causal Explanation?
Prediction asks what outcome is likely given available information. Causal research asks what would happen to the outcome under a specified change in exposure or intervention.
A predictor may be useful without causing the outcome.
For example, a laboratory measurement may indicate disease severity without being a direct cause of deterioration. Similarly, an engagement score may predict academic withdrawal while reflecting broader financial, institutional, or personal circumstances.
Changing the measured predictor does not necessarily change the underlying risk.
Consequently, a prediction model should not be interpreted as evidence that interventions targeting its predictors will improve outcomes.
Researchers should distinguish explanation and prediction in prognostic research when translating findings into recommendations.
Why Must the Intended Use Be Defined Before Modeling?
A prediction model should address a specific decision or informational need.
Researchers should identify who will use the predictions, which population will be assessed, when predictions will be generated, and what outcome horizon matters.
For example, a model estimating withdrawal risk at university enrollment requires information available at enrollment.
A model intended to update risk after the first semester may legitimately use attendance or assessment results collected during that semester.
These are different prediction tasks.
Without a defined intended use, researchers may select predictors that are statistically informative but unavailable or impractical when the model would actually be applied.
What Is the Prediction Time?
The prediction time is the point at which the model is intended to produce an estimate using information already available.
For prognostic models, this point should be aligned with a clearly defined time zero or landmark.
For example, a model intended for use at hospital admission should not rely on information obtained after several days of hospitalization.
Doing so can create information leakage and make the model appear more accurate than it would be during actual use.
Researchers should specify the predictor measurement window and ensure that each candidate predictor is available at the intended decision point.
The principles of defining time zero for prognosis are particularly relevant to future-outcome prediction.
Why Does the Prediction Horizon Matter?
A prognostic prediction requires a defined future period or outcome timescale.
For example, predicting 30-day mortality is different from predicting five-year mortality, even when both concern the same condition.
The predictor-outcome relationships, baseline event probabilities, and relevant decisions may differ across horizons.
Researchers should therefore specify the prediction horizon before selecting the model and evaluating performance.
A model developed for one horizon should not automatically be applied to another without appropriate methodological justification and validation.
Why Should Existing Prediction Models Be Reviewed First?
Prediction-model research has produced numerous models that have undergone limited independent evaluation.
Developing another model without assessing existing alternatives may contribute little to the evidence needed for practice.
Researchers should search for models addressing the same population, outcome, prediction time, and intended application.
They should assess whether the models are adequately reported, methodologically credible, and feasible to implement.
If a suitable model exists but has not been tested in the intended population, external validation may be more informative than new development.
The decision about when to develop, validate, or update a prognostic model should therefore precede the choice of algorithm.
What Makes a Dataset Suitable for Prediction Modeling?
The dataset must represent the intended population and contain reliable predictors and outcomes.
For prognostic prediction, it must also provide sufficient follow-up and an appropriate temporal relationship between predictors and outcomes.
Researchers should evaluate missingness, measurement consistency, selection mechanisms, and outcome ascertainment.
A large dataset is not automatically suitable when predictors contain future information or outcomes are systematically missing.
Similarly, a dataset with many participants but very few outcome events may be inadequate for complex model development.
Before analysis, researchers should assess whether existing data are adequate for prognostic research and whether the intended modeling strategy is supported.
How Many Predictors Should a Model Include?
The number of candidate predictors should reflect substantive knowledge, practical availability, and the amount of information in the development data.
Researchers should consider predictor parameters rather than counting only variable names.
A categorical variable, nonlinear transformation, or interaction may require several parameters.
Including too many candidate parameters relative to available information increases the risk of overfitting.
However, automatically excluding potentially useful predictors based on univariable statistical significance is not a reliable solution.
Sample size planning should use methods appropriate to the outcome and intended model complexity.
Why Is a Simple Train-Test Split Often Insufficient?
Researchers commonly divide a dataset into training and testing subsets to evaluate prediction performance.
Although this approach can be useful in particular circumstances, a single random split may be statistically inefficient when the dataset is limited.
It reduces the information available for model development and produces a test-set estimate that may be imprecise.
Bootstrapping or appropriately implemented cross-validation can provide more efficient internal validation in many settings.
However, all model-development decisions, including predictor selection and hyperparameter tuning, must be incorporated appropriately within the resampling procedure.
Internal validation does not replace external validation in independent data.
What Is Discrimination?
Discrimination concerns whether a model assigns appropriately different predictions to individuals with different outcomes.
For binary outcomes, the area under the receiver operating characteristic curve is commonly used to summarize ranking performance.
For time-to-event outcomes, discrimination measures must account for event timing and censoring.
A model with good discrimination may distinguish higher-risk from lower-risk individuals reasonably well.
However, discrimination does not establish that the predicted probabilities are numerically accurate.
Researchers should therefore avoid using AUC as the sole criterion for model quality.
What Is Calibration?
Calibration concerns agreement between predicted probabilities and observed outcome frequencies.
For example, if a model assigns approximately 20% risk to a group of individuals, the observed outcome frequency should be reasonably consistent with that prediction, allowing for sampling uncertainty.
A model may rank individuals well while systematically overpredicting or underpredicting their risks.
Such miscalibration can be consequential when decisions depend on absolute risk thresholds.
Researchers should evaluate calibration using appropriate plots and quantitative summaries.
The distinction between discrimination, calibration, and clinical usefulness is fundamental to prediction-model evaluation.
What Is Clinical or Practical Usefulness?
Predictive accuracy does not automatically establish that using a model improves decisions or outcomes.
A model may produce reasonably accurate risk estimates but add little value beyond existing assessment.
Alternatively, its predictions may trigger unnecessary interventions, consume substantial resources, or produce unequal consequences across population groups.
Decision-analytic methods can help evaluate whether predictions provide net benefit under specified assumptions and thresholds.
However, evidence of decision-analytic usefulness is not necessarily equivalent to evidence that implementing the model improves outcomes in practice.
Implementation studies may be needed when the model is intended to change clinical, educational, or organizational decisions.
Does a Prediction Model Need External Validation?
Independent external validation is generally important before broad implementation.
It evaluates the fixed model in observations not used for development and may reveal differences in calibration, discrimination, or applicability.
Validation should reproduce the intended prediction task, including predictor definitions, prediction time, and outcome horizon.
A model developed in one institution may perform differently elsewhere because of changes in population characteristics, measurement practices, or outcome frequency.
Researchers should not claim generalizability solely from strong internal validation performance.
When Is Prediction Modeling Inappropriate?
Prediction modeling may be inappropriate when the primary question concerns causal effects rather than outcome estimation.
It may also be premature when the outcome cannot be measured reliably, predictor timing is uncertain, or the available data contain insufficient information for the intended complexity.
Developing a model without a plausible use case can produce a technically sophisticated analysis with limited scientific value.
Similarly, a model intended for high-stakes decisions may be difficult to justify when its performance cannot be independently evaluated or its consequences cannot be assessed.
Researchers should consider whether a descriptive, prognostic factor, causal, or validation study would answer the underlying question more directly.
Which Reporting Guidance Should Be Used?
TRIPOD+AI provides updated reporting guidance for clinical prediction models developed or evaluated using regression and machine-learning methods.
PROBAST and its updated extensions support assessment of risk of bias and applicability in prediction-model studies.
These frameworks emphasize transparent reporting of participant selection, predictors, outcomes, sample size, modeling procedures, and performance evaluation.
Researchers should use the applicable guidance during protocol development rather than treating it as a final manuscript checklist.
For prediction studies outside clinical settings, the underlying methodological principles remain relevant even when discipline-specific reporting conventions differ.
