Can a Non-Probability Sample Support Causal Inference?

Can You Make a Causal Claim When Your Participants Were Not Randomly Sampled?

Suppose you want to determine whether AI-generated formative feedback improves students' writing performance. You recruit students from two accessible university classes because the institution has granted permission for the study.

The students were not randomly selected from the university population. Does that mean the research cannot establish whether the feedback intervention caused an improvement?

Researchers sometimes treat random sampling and random assignment as though they serve the same purpose. They do not. Random sampling primarily concerns how participants are selected from a population, while random assignment concerns how participants are allocated to treatment conditions.

A study may recruit participants non-probabilistically and still use a credible experimental design to estimate a causal effect among those participants. Conversely, a probability sample does not automatically make an observational comparison causal.

The central question is whether the research design identifies the intended causal effect and whether the sampling process permits that effect to be generalized beyond the people or settings studied.

Yes, but Causal Identification and Population Generalization Must Be Evaluated Separately

How Non-Probability Sampling Interacts with Causal Inference

Why Are Random Sampling and Random Assignment Different?

Random sampling and random assignment address different methodological problems.

Random sampling determines which eligible population units enter a study through a probability selection mechanism. It supports design-based inference about population characteristics when implemented and analyzed appropriately.

Random assignment determines which treatment condition each participating unit receives. Under a properly implemented randomized experiment, treatment assignment is independent of potential outcomes according to the randomization mechanism.

For example, researchers may recruit 100 accessible students and randomly assign 50 to receive AI-generated feedback and 50 to receive conventional feedback.

The recruitment is non-probabilistic, but the treatment assignment is randomized.

Under appropriate assumptions, the experiment may estimate the causal effect of feedback assignment for the participating students. It does not automatically estimate the average effect for every university student.

What Is the Difference Between Internal and External Validity?

Internal validity concerns whether the observed difference can credibly be attributed to the intervention or exposure rather than alternative explanations.

External validity concerns whether the findings apply to people, settings, treatments, or circumstances beyond those directly studied.

A randomized experiment conducted with an accessible sample may have strong internal validity but uncertain external validity.

For example, a well-controlled experiment among students from one university may identify an effect of AI feedback within that setting. The effect may differ in institutions with different instructional practices, language backgrounds, digital infrastructure, or student characteristics.

Non-probability recruitment therefore does not necessarily invalidate the internal causal comparison. It may, however, restrict the population to which the estimated effect can be transported.

Researchers should avoid treating internal validity and generalizability as interchangeable properties.

What Exactly Is a Causal Effect?

In the potential outcomes framework, a causal effect compares what would happen to the same unit under different treatment conditions.

For a student receiving AI-generated feedback, one potential outcome represents performance under that feedback condition, while another represents performance under the comparison condition.

Only one of those outcomes can ordinarily be observed for the same student under the same study circumstances. This is the fundamental problem of causal inference.

The sampling procedure affects which units contribute to the estimated average effect. The treatment-assignment or observational identification strategy determines whether the comparison can be interpreted causally.

Can a Convenience Sample Be Used in a Randomized Experiment?

Yes. Randomized experiments frequently recruit participants through institutions, advertisements, clinics, or other practical access routes rather than through probability sampling from a wider population.

For example, an educational technology experiment might recruit students enrolled in participating courses and randomly assign them to alternative feedback systems.

If assignment is properly randomized, outcomes are measured consistently, and relevant experimental assumptions hold, the study may support a causal estimate for the enrolled participants.

However, convenience recruitment may influence the distribution of characteristics associated with treatment effects.

If AI feedback is more effective among students with strong digital literacy, an experiment conducted primarily with digitally experienced students may produce an effect that differs from the average effect in the wider student population.

The limitations of convenience sampling should therefore be considered when defining the target of causal inference.

What Assumptions Support Causal Inference in Randomized Experiments?

Random assignment can address confounding of treatment assignment, but causal interpretation still requires attention to several conditions.

The intervention and comparison conditions should be sufficiently well defined. Outcomes should be measured appropriately, and missing outcome data should not undermine the comparison.

The consistency assumption connects observed outcomes with the potential outcomes corresponding to the treatment actually received.

Many conventional analyses also assume no interference between units, meaning one participant's treatment does not affect another participant's outcome.

That assumption may be problematic in educational settings where students collaborate, share AI-generated feedback, or exchange instructional materials.

Researchers should also distinguish the effect of assignment to an intervention from the effect of actually receiving or adhering to it.

Random assignment can identify an intention-to-treat effect under appropriate conditions, even when some participants do not comply. Estimating effects of actual treatment receipt may require additional assumptions.

Can Observational Studies with Non-Probability Samples Support Causal Inference?

Potentially, but the requirements are more demanding because treatment or exposure is not randomly assigned.

Suppose researchers recruit accessible teachers and compare those who voluntarily use generative AI with those who do not.

Teachers who adopt AI may differ in prior digital competence, motivation, institutional support, teaching assignments, or other characteristics related to the outcome.

An observed performance difference may therefore reflect confounding rather than the causal effect of AI use.

Observational causal inference requires a credible identification strategy, such as adjustment for sufficient pre-exposure confounders under conditional exchangeability, a defensible natural experiment, an instrumental variable, or another appropriate design.

Each approach has assumptions that must be justified. Statistical adjustment alone does not guarantee that all relevant confounding has been removed.

Non-probability recruitment introduces an additional consideration: conditioning on participation may itself distort relationships between exposure, outcome, and other variables.

How Can Sample Selection Create Causal Bias?

Selection bias can arise when inclusion in the analyzed sample depends on variables related to treatment and outcome.

For example, suppose a study recruits only students who voluntarily complete an optional AI learning activity. Participation may depend on both prior academic motivation and anticipated benefit from the intervention.

If the researcher compares voluntary users with nonusers, the resulting difference may reflect the selection process rather than a causal treatment effect.

Selection can also introduce collider bias when researchers condition on a variable influenced by two other variables that would otherwise not be associated in the same way.

Importantly, these problems differ from the external-validity limitation of recruiting a convenience sample before random assignment.

When recruitment precedes treatment randomization and is not affected by treatment assignment, randomization can preserve an internally valid comparison among enrolled participants. When selection occurs after treatment assignment or depends on post-treatment variables, internal causal validity may also be threatened.

What Happens When Participants Drop Out After Randomization?

Attrition can undermine causal inference if outcome availability differs systematically across treatment groups or depends on potential outcomes.

Suppose students receiving AI feedback are more likely to complete the final assessment than students receiving conventional feedback.

Comparing only students with observed final scores may no longer preserve the balance created by random assignment.

Researchers should plan strategies to minimize missing outcomes and evaluate whether missingness could bias the treatment comparison.

Depending on the design and assumptions, approaches may include intention-to-treat analysis with appropriate missing-data methods, inverse-probability weighting, multiple imputation, or sensitivity analyses.

These methods cannot guarantee recovery of the causal effect when missingness depends on unobserved outcomes in ways the available data cannot address.

The distinction between non-probability recruitment and post-randomization attrition should remain explicit.

Can a Causal Effect Be Generalized from a Non-Probability Sample?

Generalizing a causal effect requires additional assumptions about how treatment effects vary between the study participants and the target population.

Suppose an experiment estimates the effect of AI-generated feedback among students at one university. To estimate the effect for students across a national university system, researchers must consider whether relevant effect modifiers differ between the sample and the target population.

Effect modifiers are characteristics associated with differences in the magnitude or direction of treatment effects.

Potential examples include prior writing ability, language proficiency, digital literacy, instructional context, and access to technology.

Transportability methods may use population information about these characteristics to adjust or model the treatment effect for a target population.

However, their validity depends on assumptions about measured effect modifiers, overlap between study and target populations, and the correctness of the estimation approach.

A causal effect identified internally does not automatically become a population average treatment effect merely because the researcher applies statistical weights.

What Is the Difference Between a Sample Average Treatment Effect and a Population Average Treatment Effect?

The sample average treatment effect concerns the units included in the study. The population average treatment effect concerns all units in the specified target population.

Can Propensity Scores Solve Both Confounding and Sampling Problems?

Propensity scores can refer to different probabilities depending on the inferential problem.

A treatment propensity score describes the probability of receiving a treatment conditional on observed covariates. It may be used in observational causal inference under assumptions such as conditional exchangeability and positivity.

A sampling or participation propensity describes the probability of entering the study, potentially conditional on observed characteristics.

These probabilities address different selection mechanisms.

Adjusting for treatment propensity does not automatically make a non-probability sample representative of the target population. Likewise, adjusting for participation propensity does not automatically eliminate confounding between treatment and outcome.

Studies seeking both causal identification and population transportability may need to address both processes.

When Is a Causal Claim Not Defensible?

A causal claim is difficult to defend when the study cannot distinguish the intervention effect from plausible alternative explanations.

For example, a single-group pretest-posttest study of accessible students may show improved writing scores after an AI feedback activity. However, improvement could reflect practice effects, concurrent instruction, maturation, or changes in assessment difficulty.

The presence of a statistically significant difference does not establish that the intervention caused the change.

Similarly, a cross-sectional convenience survey showing an association between AI use and academic performance does not, by itself, establish temporal order or eliminate confounding.

Researchers should consider whether the design supports a causal contrast, whether assumptions are plausible, and whether alternative explanations have been adequately addressed.

When those conditions are not met, the conclusions should describe associations, experiences, or changes without attributing them confidently to a causal mechanism.

Testing AI-Generated Feedback with a Non-Probability Student Sample

Suppose the hypothetical results are:

Assume all assigned students provide valid outcome data, assignment is implemented correctly, the treatment conditions are well defined, and interference between students is negligible.

The experiment may support an internally credible causal conclusion about the effect of feedback assignment among the recruited participants.

However, the participants were not randomly selected from the wider university population. They may differ from other students in academic preparation, course requirements, digital access, or willingness to use AI.

If those characteristics modify the treatment effect, the population average effect may differ from the observed experimental estimate.

The researcher should therefore distinguish the experimental causal finding from any broader claim about all undergraduate students.

If the study instead assigned AI feedback to one existing class and conventional feedback to another without randomization, differences between classes could confound the comparison. The non-probability sample would not be the only methodological concern.

The example illustrates why causal identification and population generalization require separate evaluation.

Common Misunderstandings About Causal Inference with Non-Probability Samples

How to Evaluate Causal Claims from a Non-Probability Sample

Begin by identifying the causal effect you want to estimate and the population or group to which it refers.

Then evaluate the design's ability to identify that effect before considering generalization.

In the methodology section, describe recruitment separately from treatment assignment or exposure determination. Define the causal estimand, identify the assumptions supporting identification, and explain the population to which the conclusions apply.

Where broader generalization is attempted, report the relevant population data, adjustment procedures, and assumptions about treatment-effect heterogeneity.

The key methodological distinction is that a non-probability sample may limit where a causal finding applies without necessarily preventing the study from identifying a causal effect among those observed.

Before Making a Causal Claim, Verify the Identification and Generalization Assumptions

Frequently Asked Questions About Non-Probability Sampling and Causation

Non-Probability Recruitment Does Not Automatically Prevent Causal Identification

How to Cite This Guide

This guide is intended to be read, shared, and used in research, teaching, and academic work. If you draw on its ideas, explanations, or other content, please acknowledge the source by citing the guide. Doing so gives appropriate credit and helps your readers locate the original resource.