03 · What You Need to Know
How Synthetic Data Can Support and Complicate Prediction-Model Development
What is a prediction model?
A prediction model estimates an outcome or characteristic using information about relevant input variables.
For example, a model may estimate the probability that a student will discontinue a course based on attendance, previous academic performance, and learning-platform activity.
Prediction models may use statistical methods, machine-learning algorithms, or other computational approaches.
Their usefulness depends on whether predictions are sufficiently accurate and appropriate for the intended population and decision context.
A model that performs well on its training dataset may still perform poorly on new observations.
Consequently, model development and model validation should be treated as related but distinct activities.
How can synthetic data contribute to model development?
Synthetic data may provide additional artificial examples or controlled scenarios for training and evaluating computational procedures.
Some methods generate records resembling an empirical reference dataset, while others simulate observations under specified assumptions.
Researchers may use these records to investigate how a model behaves under particular conditions or whether its training procedure can accommodate different input patterns.
| Application |
Potential contribution |
Important limitation |
| Preliminary model development |
Testing architectures and training procedures before real data are available |
Performance on artificial data may not generalize. |
| Data augmentation |
Supplementing training examples with generated records |
Artificial patterns may introduce bias or redundancy. |
| Class imbalance research |
Exploring additional examples of underrepresented outcomes |
Generated cases may not reflect genuine rare-event characteristics. |
| Robustness testing |
Evaluating model behavior under specified variations or edge cases |
Results depend on the scenarios generated. |
| Privacy-related development |
Reducing direct use of restricted participant-level records in some workflows |
Synthetic data may still disclose sensitive information. |
| Methodological comparison |
Comparing algorithms under controlled data-generating conditions |
Conclusions remain conditional on the simulated distributions. |
The appropriate application depends on the research question and the intended use of the prediction model.
Can synthetic data compensate for a small training dataset?
Potentially, but artificial augmentation does not automatically provide new independent empirical information.
A generator trained on a small dataset may reproduce the limitations, sampling biases, or incomplete relationships present in that dataset.
Producing thousands of synthetic records from a limited empirical sample does not mean the researcher has collected thousands of additional independent observations.
However, synthetic augmentation may help certain algorithms learn useful representations or improve performance under specific conditions.
Whether this occurs must be established empirically rather than assumed from the increased number of training records.
Researchers should compare the augmented model with an appropriate baseline trained on the available genuine data.
Can synthetic data address class imbalance?
Class imbalance occurs when some outcomes or categories are represented much less frequently than others.
For example, a student dropout dataset may contain many students who completed their courses and relatively few who discontinued.
Some augmentation methods generate additional examples of underrepresented classes to support model training.
However, generated minority-class examples may not capture the full diversity of actual cases.
If the generator reproduces only a narrow set of characteristics, the prediction model may learn misleading patterns.
Researchers should assess whether augmentation improves performance on genuine observations from the underrepresented group.
They should also evaluate precision, recall, calibration, and other metrics relevant to the consequences of prediction errors.
How does synthetic-data augmentation differ from collecting more real data?
Collecting additional genuine observations may provide new empirical information about the target population.
Synthetic augmentation instead generates artificial records using a model, assumptions, or patterns learned from existing information.
The two activities are therefore not statistically equivalent.
Additional empirical observations
New information obtained through actual measurement or participation, subject to the research design and sampling process.
Synthetic augmentation
Artificial records generated from existing information or specified assumptions to support a defined computational purpose.
Artificial records may be useful during training, but they cannot independently establish that additional real-world cases were observed.
The distinction between synthetic and genuine research observations remains important when reporting sample sizes and interpreting model performance.
Can synthetic data introduce bias into prediction models?
Yes. A synthetic-data generator may reproduce biases present in its reference dataset or introduce new distortions during generation.
For example, a generator trained predominantly on records from one student population may inadequately represent students from different institutions or learning environments.
Artificial records may also underrepresent rare combinations of characteristics or produce overly simplified relationships.
These limitations can affect downstream prediction models.
A model may perform well on the majority population while producing less reliable predictions for underrepresented groups.
Researchers should therefore evaluate subgroup performance and consider whether synthetic augmentation changes disparities in prediction quality.
Statistical similarity at the overall dataset level does not guarantee adequate representation of every relevant subgroup.
What is synthetic-to-real generalization?
Synthetic-to-real generalization concerns whether a model developed using artificial observations performs effectively on genuine data from the intended application.
A model may achieve high accuracy on synthetic records because those records reflect simplified relationships or generation artifacts.
When evaluated on real observations, performance may decline if the underlying distributions differ.
This difference is sometimes described in terms of a synthetic-to-real distribution gap.
Researchers should examine whether the artificial training data preserve the predictive relationships relevant to the real target population.
Independent empirical evaluation is particularly important when predictions may influence educational, clinical, financial, or other consequential decisions.
Can a prediction model be trained entirely on synthetic data?
Technically, yes. Whether doing so is scientifically appropriate depends on the intended purpose.
A model trained entirely on synthetic data may be suitable for methodological demonstrations, software development, or evaluating behavior under a specified simulation model.
However, strong performance on synthetic evaluation data does not establish effectiveness in a real-world population.
When the intended application involves actual people or empirical outcomes, independent validation is generally necessary.
Researchers should avoid describing a model as validated for real-world use merely because it performs well on artificial records.
The evidentiary basis of the claim must remain explicit.
Why is data leakage especially important when using synthetic data?
Data leakage occurs when information that should not be available during model development influences training or model selection in a way that produces misleading performance estimates.
Synthetic-data workflows can introduce additional leakage risks when the generator is fitted using information from the eventual test set.
For example, suppose a researcher divides a genuine dataset into training and testing records only after training a synthetic-data generator on the entire dataset.
The synthetic training records may then contain information derived from observations intended to remain independent during evaluation.
Even if the final prediction model never directly receives the original test rows, the generator may have already incorporated information from them.
Researchers should establish the evaluation split before fitting data-dependent generators and ensure that test information does not influence model development.
What is the appropriate order for training and evaluation?
When the objective is to estimate predictive performance on unseen empirical data, the evaluation design should preserve independence between development and testing.
One approach is to reserve an empirical test dataset before training the synthetic-data generator.
The generator is then fitted only on the training portion, and the generated records are used for permitted model-development activities.
The final model is evaluated on the held-out empirical test set, which has not influenced generation, training, or model selection.
Where cross-validation is used, synthetic-data generation should be performed within the appropriate training folds rather than using information from validation folds.
Additional external validation may be necessary when the intended deployment population differs from the development population.
These safeguards help prevent optimistic performance estimates caused by information leakage.
How should researchers compare synthetic and real training data?
A useful evaluation compares models developed under clearly specified conditions.
For example, researchers might compare a model trained only on genuine observations with another trained on the same genuine observations supplemented by synthetic records.
Both models should be evaluated using an appropriate independent reference dataset and consistent performance metrics.
The comparison should also consider the computational cost, data-generation assumptions, and potential privacy implications of augmentation.
Researchers should not assume that a larger artificial training dataset is automatically superior.
Improvement should be demonstrated through performance relevant to the intended application.
Which performance measures should researchers examine?
The appropriate measures depend on the prediction task and the consequences of errors.
For classification, researchers may examine sensitivity, specificity, precision, recall, discrimination, and calibration.
For continuous outcomes, measures such as mean absolute error or root mean squared error may be appropriate.
However, average performance may conceal important differences across subgroups.
Researchers should also consider whether the model's predictions remain useful under changes in population characteristics or data collection conditions.
For consequential applications, model evaluation should reflect the actual decision context rather than rely exclusively on a single summary metric.
Can synthetic data improve model fairness?
Potentially, but improvement is not guaranteed.
Researchers may generate additional artificial examples for underrepresented groups or scenarios to investigate whether predictive disparities can be reduced.
However, synthetic records may reproduce stereotypes, distort subgroup distributions, or fail to represent genuine variation.
Increasing the number of generated examples does not necessarily improve fairness or reliability.
Researchers should evaluate subgroup performance on appropriate empirical data and assess whether augmentation creates new disparities.
Claims about fairness should be supported by relevant evaluation rather than inferred from the apparent balance of the synthetic training dataset.
Can synthetic data protect privacy during prediction-model development?
Synthetic data may reduce the need to expose original participant-level records during some development activities.
However, the generation model may retain information about sensitive training observations.
Generated records may closely resemble particular individuals, and some systems may be vulnerable to information-disclosure attacks.
Researchers should therefore evaluate privacy risk separately from predictive utility.
Appropriate privacy-preserving mechanisms may be necessary when source data contain sensitive information.
The question of whether synthetic data can leak participant information remains relevant even when the resulting prediction model performs well.
Can synthetic data support rare-event prediction?
Potentially. Rare outcomes may be difficult to model because few genuine examples are available.
Synthetic generation can help researchers investigate hypothetical rare-event scenarios or test how a model behaves under controlled conditions.
However, a generator cannot be assumed to reproduce the full characteristics of events that are poorly represented in its source data.
Artificial rare cases may reflect modeling assumptions more strongly than empirical evidence.
Researchers should distinguish scenario testing from demonstrating predictive accuracy for actual rare events.
Where possible, validation should include genuine cases relevant to the intended application.
How should synthetic-data use be reported in prediction-model research?
Researchers should explain the role synthetic data played in model development.
Relevant details may include the generation method, source data, training procedures, parameters, augmentation strategy, and validation approach.
The manuscript should identify which datasets were synthetic and which contained genuine observations.
Researchers should also explain how they prevented information leakage between model development and evaluation.
Performance results should be reported with sufficient detail to distinguish synthetic-data evaluation from empirical validation.
Where appropriate, reporting guidance such as TRIPOD+AI may help structure transparent reporting of prediction-model development and evaluation.
The broader issue of reporting synthetic data in research papers applies alongside prediction-model-specific requirements.
When should researchers avoid synthetic data for prediction-model development?
Synthetic data may be inappropriate when the generation process cannot adequately represent the predictive relationships required for the intended application.
They may also be unsuitable when the source dataset is too limited or biased to support credible generation, when privacy risks are unacceptable, or when independent evaluation is unavailable for consequential deployment claims.
Researchers should avoid using artificial augmentation solely to increase the apparent size of a training dataset without demonstrating its value.
The appropriate decision depends on the target population, the prediction task, the available empirical evidence, and the consequences of model errors.
Synthetic-data generation is a methodological option, not an automatic remedy for inadequate research data.
