03 · What You Need to Know
Why Population Prevalence Is Difficult to Estimate from Non-Probability Samples
What Is Population Prevalence?
Prevalence describes the proportion of a defined population possessing a specified characteristic during a defined period or at a particular point in time.
For example, a study might seek the proportion of enrolled university students who used generative AI for coursework during the previous month.
The target population, outcome definition, and reference period must be explicit.
A prevalence estimate is not simply any percentage calculated from research data. It is an estimate intended to describe a defined population.
If 144 of 200 surveyed students report using generative AI, the observed sample proportion is 72%. Whether that proportion estimates the wider student population depends on how the students entered the sample and what assumptions support the inference.
What Is the Difference Between Sample Proportion and Population Prevalence?
The sample proportion is calculated from observed participants. Population prevalence refers to the corresponding characteristic among all eligible units in the target population.
Observed sample proportion
The proportion of participating sampled units with the specified characteristic.
Population prevalence
The proportion of all eligible units in the defined target population with that characteristic.
In a properly implemented probability sample, known inclusion probabilities provide a basis for design-based estimation and sampling uncertainty.
In an ordinary non-probability sample, the selection mechanism generally does not establish those probabilities.
The observed sample proportion may therefore differ systematically from the population prevalence, even when the sample contains many respondents.
Why Does Selection Bias Matter?
Selection bias arises when the process determining who enters the observed sample is related to the outcome or to other factors that affect the estimate.
Consider an online survey about students' use of AI writing tools. Students who frequently use those tools may be more likely to notice the invitation, consider the topic relevant, and participate.
If frequent users are overrepresented, the observed proportion may exceed the population prevalence.
However, the direction of bias is not always predictable. Students who use AI in ways they believe violate institutional rules may be reluctant to disclose their behavior, potentially producing underreporting or differential participation.
Selection effects and measurement errors may operate simultaneously.
Researchers should therefore avoid assuming that non-probability sampling necessarily overestimates or underestimates prevalence. The direction and magnitude depend on the actual selection and measurement processes.
Why Does a Large Non-Probability Sample Not Solve the Problem?
Increasing sample size can reduce some forms of random variability, but it does not automatically reduce systematic selection bias.
A survey with 10,000 voluntary respondents may still exclude students with limited internet access or those who rarely engage with institutional communication channels.
If these exclusions are associated with the outcome, the observed prevalence may remain distorted despite the large sample.
Meng's analysis of large datasets emphasizes that data quantity alone does not guarantee accurate population inference when selection mechanisms create systematic differences between observed and unobserved units.
The relevant issue is not simply how many people responded, but how the responding sample relates to the target population.
Can Weighting Correct a Non-Probability Prevalence Estimate?
Weighting may improve a non-probability estimate when reliable population information is available and the adjustment variables adequately address relevant selection differences.
For example, a university may know the population distribution by academic program, year level, and institutional sector.
A researcher could construct weights that align the responding sample with those known distributions.
However, matching observed characteristics does not guarantee that respondents and nonrespondents are similar on the outcome within each adjustment category.
Students with high AI engagement may remain overrepresented within every academic program and year level.
Weighting is therefore a model-dependent adjustment rather than proof that the sample has become representative.
How Does Poststratification Work?
Poststratification uses known population counts for defined categories to adjust the contribution of sample observations.
For a population divided into mutually exclusive categories, a poststratified prevalence estimate can be calculated as the population-weighted average of category-specific sample proportions.
Poststratification requires suitable population benchmarks and adequate observations in the relevant categories.
If important categories have no respondents, their outcome prevalence cannot be directly estimated from the sample without additional modeling or information.
Even when every category contains respondents, selection bias may remain within categories.
What Assumptions Make Non-Probability Adjustment Plausible?
Many approaches to non-probability population estimation rely on assumptions that observed auxiliary variables adequately explain differences between sampled and nonsampled units.
One common formulation assumes that, conditional on the adjustment variables, sample participation is independent of the outcome being estimated.
This assumption is sometimes described as conditional exchangeability or an ignorable selection mechanism.
Another important condition is positivity: population groups relevant to the estimation procedure must have a nonzero opportunity for observation under the assumed model.
For example, if students without internet access have no opportunity to enter an online sample, an adjustment method cannot directly learn their AI-use prevalence from that sample.
Model-based extrapolation may still be attempted, but it requires additional assumptions and increases uncertainty.
These conditions are often difficult to verify completely because outcomes for nonsampled individuals are not observed.
What Is Propensity-Based Adjustment?
Propensity-based approaches attempt to model the likelihood of inclusion in a non-probability sample using observed characteristics.
Some methods combine a non-probability sample with a suitable probability reference sample containing common auxiliary variables.
The researcher may estimate relative participation propensities or construct weights intended to reduce observed differences between the two samples.
Such approaches require careful attention to the reference sample's design, the comparability of variables, model specification, and assumptions about selection.
Simply fitting a regression model predicting participation does not establish that the resulting weights remove all selection bias.
Chen, Li, and Wu discuss approaches for population inference using non-probability samples and auxiliary probability-sample information, including the assumptions and estimation considerations involved.
What Is Calibration Weighting?
Calibration weighting adjusts sample weights so that weighted totals match specified, reliable population benchmarks.
For example, weights might be calibrated to known totals for student level, academic program, and geographical location.
This can reduce discrepancies on the calibration variables and may improve estimates when those variables are related to both participation and the outcome.
However, calibration does not automatically correct differences involving unobserved characteristics.
Extreme weights can also increase estimator variability, particularly when certain population categories are represented by very few respondents.
Researchers should examine the distribution of weights and conduct sensitivity analyses rather than assuming that calibration always improves accuracy.
Can Statistical Models Estimate Prevalence from Non-Probability Samples?
Yes, under suitable assumptions.
Outcome models can estimate prevalence as a function of observed characteristics and then combine predictions using known population distributions.
For example, a researcher might model generative AI use according to academic program, year level, and institutional sector, then aggregate predicted probabilities across the known population composition.
Multilevel regression and poststratification is one approach used for this type of population estimation, although its suitability depends on the available data, model structure, and target population.
Model-based estimates are sensitive to misspecification and unmeasured selection differences.
Researchers should distinguish model-based uncertainty from the uncertainty arising from unverified selection assumptions. A narrow model-based confidence interval does not necessarily capture all possible selection bias.
Can You Calculate a Margin of Error for a Non-Probability Sample?
Conventional design-based margins of sampling error depend on a known probability selection mechanism.
For an ordinary convenience, quota, or snowball sample, applying the simple random sampling margin-of-error formula does not establish a valid probability-sampling interpretation.
For example, the familiar expression involving 1.96 multiplied by the square root of p(1 − p) / n assumes a particular sampling and estimation framework.
Using that formula for a voluntary online sample can create a misleading impression of precision because it does not account for unknown selection differences.
Model-based standard errors or intervals may be calculated under specified assumptions, but they should be identified as such.
Researchers should also consider sensitivity analysis to examine how estimates change under plausible alternative assumptions about selection.
How Can Sensitivity Analysis Improve Interpretation?
Sensitivity analysis evaluates how conclusions depend on uncertain assumptions or analytical choices.
For a non-probability prevalence estimate, researchers might compare unadjusted results with estimates using alternative weighting variables, outcome models, or population benchmarks.
They may also examine how estimates change when plausible differences between respondents and nonsampled individuals are introduced.
Agreement across several adjustments may increase confidence in the stability of an estimate under those approaches, but it does not prove that unobserved selection bias is absent.
Disagreement can reveal that the estimate is sensitive to modeling decisions and should be interpreted cautiously.
Transparent reporting of these analyses is more informative than presenting one adjusted percentage as though it were assumption-free.
When Should Researchers Avoid Population Prevalence Claims?
Population prevalence claims are difficult to defend when the target population is poorly defined, recruitment coverage is highly restricted, population benchmarks are unavailable, or the selection mechanism is strongly related to the outcome in ways that cannot be addressed credibly.
For example, an open social media survey about generative AI use may attract participants with unusually strong interest in the topic.
If the researcher lacks information about the people who did not encounter or respond to the invitation, population prevalence estimation may depend on assumptions that cannot be adequately justified.
In such circumstances, it is more defensible to report the observed respondent proportion and explain the limits of generalization.
The fact that a prevalence estimate can be calculated mathematically does not mean that it is methodologically supported.
How Should Non-Probability Prevalence Estimates Be Reported?
Researchers should identify the target population, recruitment mechanism, sample size, outcome definition, and population information used for adjustment.
Where weighting or modeling is applied, report the adjustment variables, data sources, assumptions, and uncertainty estimation procedures.
It is often useful to present the unadjusted respondent proportion alongside the adjusted estimate, clearly distinguishing their interpretations.
Researchers should also report important limitations, including restricted coverage, potential unmeasured selection differences, and uncertainty not fully represented by model-based intervals.
The AAPOR Task Force on Non-Probability Sampling provides methodological guidance on evaluating non-probability survey estimates and their underlying assumptions.
