03 · What You Need to Know
The Scientific Conclusions That AI-Simulated Participants Cannot Establish on Their Own
Why do scientific claims depend on the origin of the observations?
Scientific inference requires a defensible relationship between the observations collected and the phenomenon being investigated.
When researchers collect questionnaire responses from actual students, those responses provide information about the participating individuals under the conditions of the study.
When researchers generate hypothetical student responses through a language model, the observations instead describe how the model responded to particular instructions.
These two forms of evidence may contain similar variables and numerical values, but their origins differ.
The difference affects what researchers can conclude.
A model-generated response may support a finding about computational behavior without independently establishing a corresponding fact about human behavior.
Which scientific claims cannot be supported by simulated participants alone?
| Unsupported claim |
Why simulation alone is insufficient |
Additional evidence needed |
| Actual population prevalence |
Generated records are not observations from a human sampling frame. |
Appropriate empirical population evidence or independently validated estimation methods |
| Genuine lived experiences |
AI-generated narratives are not accounts supplied by people who experienced the events. |
Authentic participant testimony or other relevant empirical evidence |
| Real-world causal effects |
Effects on model outputs do not automatically transfer to human behavior. |
An appropriate empirical causal identification strategy |
| Demographic group differences |
Prompted personas may reproduce assumptions or stereotypes. |
Empirical subgroup comparisons with appropriate measurement and sampling |
| Human questionnaire validity |
Artificial response patterns do not establish how people interpret or respond to an instrument. |
Relevant human measurement and validation evidence |
| Behavioral mechanisms |
Matching an outcome does not establish the process responsible for human behavior. |
Theoretical and empirical evidence addressing the proposed mechanism |
| Population generalizability |
Model outputs do not automatically represent the target population. |
Validation across appropriate populations, contexts, and conditions |
These limitations do not make simulation research scientifically worthless. They define the boundary between what was observed and what remains to be established.
Can simulated participants establish what real people believe?
Not independently.
A language model may generate statements expressing support for a policy, dissatisfaction with an educational platform, or preferences for a teaching method.
However, these statements are produced by the model rather than reported by actual members of the population.
For example, 700 generated responses expressing support for blended learning do not establish that 70 percent of university students support that approach.
The proportion describes the simulated dataset.
Researchers may investigate whether such predictions correspond to independent survey evidence, but they should not present the generated proportion as a directly observed population estimate.
Can AI-generated interviews establish lived experiences?
No. A generated narrative cannot independently establish that a person experienced the circumstances described.
For example, an AI system may produce a detailed account of a teacher struggling to implement digital learning tools in a rural school.
The narrative may contain plausible descriptions of connectivity problems, institutional constraints, and instructional adjustments.
However, it is not testimony from an actual teacher.
Qualitative researchers should therefore avoid treating fictional narratives as evidence of lived experience, even when they can be analyzed using established qualitative methods.
The limitations of AI-simulated qualitative interviews are particularly consequential when research aims to represent participants' perspectives.
Can statistical significance overcome the limitations of synthetic participants?
No. Statistical significance does not change the origin of the observations.
Researchers can calculate p-values, confidence intervals, and regression coefficients from generated datasets.
However, the interpretation of those statistics depends on the data-generating process and the assumptions of the analytical method.
For example, a statistically significant relationship between instructor support and engagement in 10,000 AI-generated responses may reflect associations learned by the model or introduced through prompting.
It does not independently establish that the relationship exists among actual students.
Furthermore, repeated model outputs may share dependencies that make conventional human-sampling assumptions inappropriate.
Statistical precision within a simulation should not be confused with empirical validity outside it.
Can simulated participants establish causal effects in humans?
Not without additional empirical justification.
A researcher may expose AI-generated respondents to different hypothetical interventions and compare their responses.
For example, a language model might produce higher engagement ratings after receiving a motivational message.
Under an appropriate experimental design, the researcher may estimate the effect of that message on the model's outputs.
However, this does not establish the effect the message would have on actual students.
Human responses may depend on social context, incentives, prior experiences, and other mechanisms not adequately represented by the simulation.
Researchers should distinguish causal effects within the computational system from causal effects in the intended human population.
Can AI-generated respondents establish demographic differences?
They can reveal how a model responds to demographic prompts, but not independently how actual demographic groups differ.
For example, a model may generate lower engagement ratings when instructed to portray students from a particular socioeconomic background.
That pattern may reflect stereotypes or assumptions in the prompt or model training.
It should not be interpreted as evidence that real students from that background have lower engagement.
Researchers should examine whether demographic conditioning introduces unsupported associations.
Empirical evidence is necessary when conclusions concern actual differences among human groups.
Can AI-generated participants establish questionnaire reliability?
Not for the intended human population on their own.
A researcher may calculate Cronbach's alpha or another reliability coefficient from artificial questionnaire responses.
Such calculations describe the generated data under the relevant assumptions.
However, they do not establish that actual respondents would interpret the items consistently or produce scores with similar measurement properties.
A model may generate coherent response patterns because the prompts encourage consistency across related concepts.
Researchers should therefore avoid reporting synthetic reliability estimates as evidence that a questionnaire has been validated among real participants.
Can simulated participants establish construct validity?
No, not independently.
Evidence supporting the validity of questionnaire score interpretations may involve response processes, internal structure, relationships with other variables, and consequences of use.
AI-generated responses can help test analytical procedures or explore how a model represents theoretical constructs.
However, a factor structure obtained from artificial responses does not establish that genuine respondents interpret the instrument according to the intended construct model.
The distinction between LLM-generated responses and human survey data remains essential when interpreting psychometric analyses.
Can AI-simulated participants establish the prevalence of a behavior?
No, not through artificial response counts alone.
For example, generating 1,000 hypothetical students and observing that 300 report academic procrastination does not establish a 30 percent prevalence rate among actual students.
The generated proportion depends on the model and its instructions.
Researchers may use simulation to investigate whether a validated predictive method approximates known prevalence estimates.
However, the artificial sample itself does not constitute direct population measurement.
Claims about prevalence require appropriate empirical or separately justified model-based evidence.
Can AI simulations establish human behavioral mechanisms?
Not simply by reproducing an expected outcome.
For example, a language model may generate responses consistent with a theory that perceived instructor support increases motivation.
However, reproducing that association does not demonstrate that the model represents the psychological processes through which instructor support affects actual students.
The result may arise from learned linguistic associations or assumptions embedded in the simulation.
Researchers should distinguish reproducing an outcome pattern from identifying the mechanism responsible for that pattern.
Human behavioral mechanisms require appropriate theoretical and empirical investigation.
Can simulated participants establish that an intervention works?
Not for actual participants without appropriate empirical evidence.
For example, an AI simulation may suggest that personalized feedback improves hypothetical student motivation.
However, the generated improvement does not demonstrate that the intervention would produce the same result in a classroom.
Real-world effectiveness may depend on implementation quality, participant characteristics, institutional conditions, and other contextual factors.
Simulation can help formulate hypotheses or test procedures before intervention research.
It should not be presented as an empirical effectiveness trial involving human participants.
Can AI-generated responses establish qualitative saturation?
Not in the same sense as evaluating the adequacy of genuine qualitative data collection.
A language model may generate many fictional interviews until additional outputs appear to contain few new themes.
However, this pattern may reflect repetition within the model's learned representations or prompting procedure.
It does not establish that additional human interviews would yield no relevant new information.
Moreover, saturation is not a universal quality criterion across all qualitative methodologies.
Researchers should evaluate information adequacy according to the specific qualitative approach and the empirical material collected.
Can AI-generated participants establish cultural representativeness?
Not automatically.
A language model may produce responses in a particular language or adopt a hypothetical cultural profile.
However, linguistic fluency does not establish accurate representation of local experiences, values, or social practices.
Models may reproduce generalized descriptions that overlook differences within communities.
Researchers should avoid treating AI-generated cultural personas as substitutes for engagement with the relevant population.
Claims about cultural practices require appropriate contextual and empirical evidence.
Can AI-generated participants establish current public opinion?
Not independently.
Population attitudes may change in response to recent events, policies, and social conditions.
A language model may rely on historical training patterns or contextual information supplied through prompts.
Even when the outputs appear current, their correspondence with actual public opinion requires validation.
For example, an AI-generated survey about a newly implemented educational policy cannot establish how teachers currently perceive that policy without relevant empirical evidence.
The problem is especially important when attitudes change more quickly than the information available to the model.
Does agreement among AI models establish human truth?
No. Agreement across models may indicate consistency among the systems evaluated, but it does not independently establish correspondence with human reality.
Different models may share training sources, design assumptions, or systematic biases.
For example, several models might generate similar narratives about students experiencing online learning difficulties.
Their agreement could reflect common patterns in training material rather than independent observations of student experiences.
Researchers should distinguish cross-model consistency from empirical confirmation.
Can AI-simulated participants establish that a study is generalizable?
Not solely through the number or diversity of generated profiles.
Generalizability concerns whether findings apply beyond the conditions examined.
Generating hypothetical participants from many demographic categories does not establish that the simulation accurately represents those groups.
Researchers should evaluate relevant populations, contexts, prompts, and model versions using appropriate evidence.
Even strong performance on one empirical benchmark may not transfer to other settings.
Claims about generalization should specify the conditions under which the evidence was obtained.
Can researchers make any claims about humans from AI simulations?
Potentially, when an appropriate empirical validation framework supports the inference.
For example, a study may establish that a model predicts selected human survey distributions with a specified level of accuracy on independent benchmark data.
That finding supports a claim about predictive correspondence under the evaluated conditions.
However, the evidence does not automatically establish universal representativeness or replace direct measurement of the population.
The distinction between scientific claims supported by AI-simulated participants and unsupported generalizations depends on the scope of validation.
Researchers should state the observed agreement and its limitations rather than converting a bounded prediction result into a broad statement about human behavior.
Why is epistemic circularity a concern?
Epistemic circularity occurs when the assumptions used to generate simulated responses are subsequently presented as though they were independently discovered findings.
For example, a researcher may instruct an AI model that students with low instructor support tend to have low engagement.
If the generated dataset subsequently shows that relationship, the result may largely reproduce the initial instruction.
Statistical analysis cannot resolve the circularity because the expected relationship was embedded in the data-generation process.
Researchers should distinguish assumed relationships from outcomes independently established through observation or validation.
This is particularly important when simulations are used to investigate theoretical propositions.
How should unsupported claims be corrected?
Researchers should identify the actual source of the observations and revise conclusions to reflect what the simulation demonstrates.
For example, a statement that university students prefer blended learning may need to be revised to indicate that a particular model generated a preference for blended learning under specified prompts.
Where the intended conclusion concerns humans, researchers should obtain appropriate empirical evidence or present the result as an unvalidated hypothesis.
Artificial sample sizes should be reported separately from genuine participant counts.
Methods and limitations should explain the generation process and relevant uncertainties.
The objective is not to weaken a valid finding unnecessarily, but to ensure that its scope matches the evidence.
How should limitations be reported in a manuscript?
Researchers should explain the boundaries of the simulation and their implications for scientific interpretation.
Relevant limitations may include prompt sensitivity, model dependence, training-data bias, demographic representation, response variability, and lack of independent human validation.
However, limitations should be specific rather than reduced to a generic statement that AI may be inaccurate.
The discussion should identify which conclusions remain supported and which require further empirical investigation.
Clear reporting helps readers distinguish a legitimate study of model behavior from an unsupported claim about actual participants.
