03 · What You Need to Know
Why Choosing a Final Statistical Model Requires More Than Ranking Performance Scores
What Does Selecting a Final Statistical Model Mean?
Statistical model selection involves choosing a model specification from a set of possible alternatives for a defined analytical purpose.
Alternatives may differ in their predictors, functional forms, interactions, distributional assumptions, estimation methods, or treatment of missing observations.
For example, a researcher might compare ordinary least squares regression, a model incorporating nonlinear terms, and a multilevel model accounting for students nested within classes.
These models may answer related but different questions. They may also rely on different assumptions about how observations were generated and how uncertainty should be estimated.
The final model is ordinarily the specification used for the primary reported analysis. It should be distinguished from exploratory models, robustness checks, and sensitivity analyses.
Researchers should not assume that every study must produce one uniquely correct model. Multiple specifications may provide useful perspectives on the same research problem.
Model Selection Depends on Whether the Goal Is Prediction or Inference
A fundamental distinction concerns the purpose of the analysis.
| Analytical objective |
Primary consideration |
Implication for model selection |
| Prediction |
Performance on relevant new observations |
Evaluate generalization, calibration, prediction error, and deployment conditions |
| Causal inference |
Identification and estimation of a specified causal effect |
Prioritize study design, causal assumptions, and an appropriate adjustment strategy |
| Explanatory analysis |
Understanding theoretically meaningful relationships |
Consider theoretical coherence, measurement, assumptions, and interpretability |
| Descriptive analysis |
Summarizing patterns or associations in a defined population |
Choose specifications that accurately represent the intended descriptive quantity |
A model optimized for prediction may use variables that are inappropriate for estimating a causal effect. Conversely, a causal model may deliberately omit variables that would improve prediction but alter the intended estimand.
AI systems that rank models without identifying the analytical objective may therefore recommend an unsuitable specification.
The Highest R-Squared Does Not Automatically Identify the Best Model
R-squared describes the proportion of observed outcome variation accounted for by a fitted model under the relevant definition. In ordinary least squares regression with an intercept, adding predictors cannot reduce the unadjusted R-squared.
Consequently, a more complex model may have a higher R-squared even when the additional predictors contribute little meaningful information.
Adjusted R-squared incorporates a penalty related to model complexity, but it does not establish causal validity, correct model specification, or generalizability.
Researchers should avoid selecting models solely because they explain the largest proportion of variation in the observed sample.
A model may fit the existing data closely while performing poorly on new observations or providing misleading estimates of the relationships under investigation.
AIC and BIC Answer Particular Model-Comparison Questions
Information criteria offer systematic ways to compare certain fitted models while accounting for model complexity.
The Akaike Information Criterion, or AIC, is commonly expressed as follows:
A lower AIC indicates a more favorable value of that criterion within a valid comparison. It does not mean that Model B is true, causally appropriate, or guaranteed to perform best in every setting.
The Bayesian Information Criterion, or BIC, uses a different complexity penalty:
The disagreement reflects the different penalties and objectives of the criteria. Neither criterion provides a universal answer to which model should become the final scientific analysis.
Information criteria must also be compared under appropriate conditions, including compatible likelihood definitions and relevant observations. Values from incompatible model-fitting procedures should not be ranked indiscriminately.
Cross-Validation Helps Evaluate Prediction but Does Not Establish Causality
Cross-validation estimates aspects of predictive performance by repeatedly separating data used for model fitting from data used for evaluation.
For example, in k-fold cross-validation, the dataset is divided into k subsets. Models are fitted using all but one subset and evaluated on the held-out subset, repeating the procedure across folds.
Cross-validation can help compare predictive models and assess whether apparent performance is likely to generalize beyond the training observations.
However, its reliability depends on an appropriate resampling design. Ordinary random folds may be unsuitable when observations are clustered, temporally ordered, or otherwise dependent.
Feature selection, preprocessing, and hyperparameter tuning must also be incorporated correctly into the training procedure to avoid leakage.
Good cross-validated prediction does not establish that a model identifies causal relationships.
Overfitting Can Make an AI-Selected Model Look Better Than It Is
Overfitting occurs when a model captures sample-specific variation that does not generalize adequately to new observations.
Complex models may be particularly susceptible when the available data contain limited information relative to the number of estimated parameters.
AI-assisted model searches can intensify this concern when many specifications are evaluated and the most favorable result is selected.
Even if each candidate model is fitted correctly, the process of searching across many alternatives may produce an overly optimistic assessment of the selected model.
Researchers should distinguish performance measured on the same data used for selection from performance evaluated through an appropriate independent procedure.
For prediction, nested cross-validation or a properly reserved test set may help evaluate the complete model-selection process.
Model Assumptions Must Be Evaluated, Not Merely Reported
Statistical models rely on assumptions concerning the data-generating process, functional relationships, dependence, and error structure.
In linear regression, relevant considerations may include the functional form of the conditional mean, independence or dependence of observations, the treatment of heteroskedasticity, and assumptions required for particular inferential procedures.
Researchers should avoid the common misconception that every observed variable must be normally distributed for linear regression to be valid.
For generalized linear models, appropriate distributional and link-function assumptions become relevant. Multilevel models introduce additional considerations about hierarchical structure and random effects.
AI may suggest diagnostics, but researchers should verify whether those diagnostics are appropriate for the selected model and whether their interpretation is correct.
Passing a diagnostic test does not prove that all model assumptions hold.
Covariate Selection Can Change the Meaning of the Model
Choosing a final model often involves deciding which variables to include.
For causal analysis, adding or removing covariates can change the adjustment strategy and potentially the causal quantity being estimated.
A variable may be a confounder, mediator, collider, or another component of the assumed causal structure. Its statistical association with the outcome does not determine whether adjustment is appropriate.
Researchers should therefore distinguish defensible covariate selection from automated feature selection intended to maximize predictive performance.
A model with fewer statistically significant predictors may still be preferable when it better corresponds to the research design and intended estimand.
Post-Selection Inference Can Be Misleading
When researchers select a model after examining many alternatives, conventional standard errors, confidence intervals, and p-values may not account for the selection process.
For example, fitting numerous models and reporting only the one with the smallest p-value can increase the risk of misleading inference.
This concern is related to multiple testing, selective reporting, and researcher degrees of freedom.
Model selection is not inherently inappropriate. The problem arises when inferential claims are made as though the final specification had been selected independently of the observed results.
Researchers should prespecify confirmatory analyses where feasible and disclose exploratory model searches.
Depending on the analytical objective, suitable approaches may include sample splitting, appropriate selective-inference procedures, independent validation, or clearly qualified exploratory reporting.
Unusual Observations Should Not Be Removed Merely to Improve Model Fit
AI may recommend removing observations that produce large residuals or substantially affect regression coefficients.
However, influential observations can be valid and scientifically informative.
Researchers should investigate whether unusual values reflect measurement errors, legitimate variation, or model misspecification.
The principles governing AI-assisted statistical outlier removal and observation exclusion decisions should be applied independently of whether the resulting model fits better.
Removing valid observations to improve statistical significance or diagnostic appearance can compromise the intended analysis.
Interpretability and Scientific Coherence Matter
Some models achieve favorable predictive performance while being difficult to interpret.
This may be acceptable when the primary goal is prediction and the model can be evaluated appropriately. It may be less suitable when the research question concerns a specific theoretical relationship or causal mechanism.
Interpretability is not a single property. It may concern the meaning of parameters, the transparency of assumptions, the ability to explain predictions, or the reproducibility of analytical decisions.
Researchers should specify which form of interpretability matters for their study.
AI-generated explanations of complex models should not be assumed to faithfully describe the model's actual computations or establish causal mechanisms.
The Final Model Should Not Be Chosen to Confirm the Hypothesis
A model that produces the expected direction or statistical significance is not necessarily more valid than one that contradicts the researcher's hypothesis.
Researchers should avoid changing covariates, transformations, exclusion criteria, or functional forms until the desired result appears.
Such practices may create an appearance of stronger evidence than the analytical process supports.
The question of whether a result supports the research hypothesis should be addressed after evaluating the design, assumptions, estimates, and uncertainty.
It should not determine which model is selected.
Automated Model Selection Can Be Appropriate Under Defined Conditions
Automated model selection is not inherently invalid.
For example, a predictive study may use a prespecified algorithm to compare candidate models through an appropriate nested cross-validation procedure.
The automated procedure can be scientifically defensible when its objective, candidate models, evaluation metric, and validation strategy are justified in advance.
Similarly, automated optimization may be appropriate for selecting tuning parameters within a properly designed modeling pipeline.
However, the research team remains responsible for determining whether the selected model is appropriate for the intended use and whether the evaluation supports the claims being made.
Automation should implement a defensible analytical strategy rather than invent the strategy after examining the desired results.
Researchers Should Preserve the Model-Selection Record
Transparent reporting should identify the candidate models considered, the selection criteria, the role of AI, and any departures from the original analytical plan.
Researchers should preserve relevant code, data-processing procedures, software versions, and model settings where permitted.
When the final model is sensitive to plausible alternative specifications, that dependence should be acknowledged.
For predictive modeling, reporting guidance such as TRIPOD+AI may be relevant in its applicable domain. Other disciplines and study designs may require different reporting standards.
Watch Out
Do not ask AI to search for the model that produces the strongest statistical significance and then report that model as though it were specified before examining the data. This can undermine the validity of conventional inferential claims.
