03 · What You Need to Know
How Selection Probabilities Determine Sampling Weights
What Is an Inclusion Probability?
An inclusion probability is the probability that a particular eligible population unit will be included in the selected sample under the specified sampling design.
In probability sampling, eligible units must have known, nonzero inclusion probabilities for the intended design-based inference.
Under simple random sampling without replacement, every population unit has the same inclusion probability, calculated as the sample size divided by the population size.
More complex designs may deliberately assign different probabilities to different units.
For example, researchers may oversample a small population group, select larger institutions with greater probability, or use several selection stages in which the overall probability depends on successive conditional selections.
Unequal probabilities are not inherently problematic. They become problematic when the resulting estimates ignore the sampling mechanism.
What Is a Base Sampling Weight?
A base sampling weight, also called a design weight, is the reciprocal of a population unit's inclusion probability.
The interpretation is statistical rather than personal. A larger base weight compensates for a lower probability of inclusion under the sampling design.
Why Do Unequal Selection Probabilities Arise?
Unequal probabilities commonly arise from deliberate features of probability sampling.
In disproportionate stratified sampling, different strata are sampled at different rates. This may be necessary to obtain adequate observations from small population groups.
In probability proportional to size sampling, larger clusters may have greater selection probabilities than smaller clusters.
In multistage sampling, the overall inclusion probability may depend on the probability of selecting a geographical area, an institution within that area, and an individual within that institution.
Unequal probabilities can also arise when researchers use different sampling fractions across geographical domains or select certain units with certainty.
The reason for unequal selection should be documented because it influences both the calculation and interpretation of weights.
How Are Weights Calculated in Stratified Sampling?
When simple random sampling without replacement is conducted independently within each stratum, the inclusion probability equals the stratum sample size divided by the stratum population size.
Under exact proportional allocation, the sampling fractions and base weights would be equal across strata. Under disproportionate allocation, they generally differ.
How Are Overall Inclusion Probabilities Calculated in Multistage Sampling?
In a multistage design, the overall inclusion probability is obtained by combining the relevant conditional selection probabilities across the stages.
For a unit with one eligible selection path through a three-stage design, the calculation is:
Where units can be reached through overlapping frames or multiple selection paths, inclusion probabilities require additional calculations. Simply multiplying one sequence of probabilities may be insufficient.
How Do Sampling Weights Affect Population Totals?
One foundational approach to design-based estimation is the Horvitz-Thompson estimator, which uses inverse inclusion probabilities to estimate a population total.
In practical surveys, weights may subsequently be adjusted for nonresponse or calibrated to reliable population totals. Those procedures produce adjusted weights rather than the original base design weights.
How Do You Calculate a Weighted Population Mean?
A common weighted mean divides the sum of weighted observations by the sum of the weights.
This ratio estimator is widely used for survey means and proportions. Unlike the Horvitz-Thompson total estimator, a ratio estimator is not generally exactly design-unbiased in finite samples, although it can have favorable large-sample properties under appropriate conditions.
If the population size N is known, the Horvitz-Thompson estimate of the mean can alternatively be calculated by dividing the estimated total by N.
What Is the Difference Between Base Weights and Final Survey Weights?
Base weights represent the original probability selection mechanism. Final survey weights may incorporate additional adjustments.
| Weight type |
Purpose |
Important limitation |
| Base design weight |
Accounts for unequal inclusion probabilities |
Does not correct nonresponse or coverage errors |
| Nonresponse-adjusted weight |
Adjusts for differential response using specified information or models |
Depends on assumptions about response behavior |
| Calibrated weight |
Aligns weighted sample totals with reliable external population totals |
Cannot guarantee correction of unobserved differences |
| Normalized weight |
Rescales weights to a chosen total while preserving relative weights |
May be unsuitable for estimating population totals directly |
Researchers should distinguish these weights in their methodology and data documentation.
For example, if a dataset contains a final calibrated survey weight, replacing it with a newly calculated inverse selection probability may discard important adjustments made by the survey producer.
Conversely, treating an unadjusted base weight as though it corrected all forms of survey error would overstate its capabilities.
Do Unequal Weights Affect Statistical Precision?
Yes. Unequal weights can increase the variability of weighted estimates, although the actual effect depends on the relationship between weights, outcomes, and the sampling design.
One commonly used approximation for the weighting component of a design effect is based on the coefficient of variation of the weights.
The approximation is often associated with Kish's unequal-weighting effect. It should not be treated as an exact variance formula for every estimator or complex survey design.
Researchers should use design-aware variance estimation rather than relying solely on weight variability to characterize uncertainty.
What Is the Effective Sample Size Under Unequal Weights?
A commonly used weight-based effective sample size approximation is:
Effective sample size is estimator- and design-dependent. A weight-only approximation does not account fully for clustering, stratification, or relationships between weights and study outcomes.
The implications for precision and sample size planning should therefore be considered alongside the broader design effect.
Can Sampling Weights Correct Selection Bias?
Weights can account for known unequal selection probabilities under a valid probability design. This is different from correcting every form of selection bias.
For example, if certain eligible population units had no opportunity for selection because they were missing from the sampling frame, inverse-probability weights calculated from the available frame do not automatically represent those omitted units.
Likewise, if selected participants do not respond, their response behavior may introduce systematic differences between respondents and nonrespondents.
Nonresponse adjustment may reduce bias when appropriate auxiliary information and assumptions are available, but it cannot guarantee correction when response differences are related to unobserved outcomes.
Researchers should distinguish design-based weighting from model-dependent adjustments intended to address incomplete response or coverage.
Should Every Statistical Analysis Use Sampling Weights?
There is no universal rule that every analysis must use weights in exactly the same way.
For design-based estimation of population totals, means, and proportions, accounting for unequal inclusion probabilities is generally essential.
For regression and other model-based analyses, the appropriate treatment of weights depends on the estimand, sampling mechanism, model assumptions, and inferential framework.
Some analyses require survey-weighted estimation, while others may use design variables, conditional models, or alternative procedures under defensible assumptions.
Researchers should not automatically discard weights because an unweighted model produces smaller standard errors or more favorable statistical results.
Nor should they assume that applying weights to an ordinary statistical procedure automatically produces valid standard errors.
The estimation and variance procedures must be compatible with the sampling design and the research question.
04 · A Practical Example
Estimating AI Adoption After Oversampling Private-School Teachers
Hypothetical Example
A Stratified Survey with Unequal Selection Probabilities
A researcher wants to estimate the proportion of teachers using generative AI for instructional planning across a population of 10,000 teachers.
The population contains 8,000 public-school teachers and 2,000 private-school teachers. To support a sector comparison, the researcher selects 400 teachers from each group through independent simple random sampling without replacement.
Because the two strata differ in population size but contribute equal selected sample sizes, their inclusion probabilities and base weights differ.
| Characteristic |
Public schools |
Private schools |
| Population size |
8,000 |
2,000 |
| Selected sample |
400 |
400 |
| Inclusion probability |
5% |
20% |
| Base design weight |
20 |
5 |
| Hypothetical AI adoption rate |
40% |
60% |
Assume complete response and no additional adjustments. The unweighted pooled estimate is:
(400 × 0.40 + 400 × 0.60) / 800 = 0.50, or 50%.
However, this estimate gives equal influence to the two sectors even though public-school teachers constitute 80% of the population.
The weighted estimate accounts for the actual population shares.
The researcher should calculate confidence intervals using a procedure that accounts for the stratified design rather than treating the weighted observations as independent replicated population records.
If response rates differ between sectors, additional nonresponse adjustments may be considered using defensible auxiliary information.
This example illustrates the central principle: unequal selection probabilities are compatible with rigorous probability sampling, but they must be reflected in the estimation procedure when the objective is to describe the overall population.
06 · What This Means for You
How to Handle Unequal Selection Probabilities in Your Research
Begin by reconstructing the actual probability selection mechanism. Determine each selected unit's inclusion probability before calculating its base weight.
Then identify the population quantity you intend to estimate and select an estimation procedure compatible with the design.
A simple decision framework
If all eligible units have equal inclusion probabilities
Base weights are constant, although other adjustments may still be necessary.
If strata are sampled at different rates
Calculate stratum-specific inclusion probabilities and corresponding inverse-probability weights.
If selection occurs through several sampling stages
Calculate overall inclusion probabilities from the relevant conditional selection procedures.
If some selected units do not respond
Evaluate justified nonresponse adjustments rather than assuming base weights already correct response bias.
If weights vary substantially
Examine their distribution, potential influence on estimates, and implications for statistical precision.
If the survey also involves stratification or clustering
Use an appropriate survey-design analysis that accounts for the relevant features when estimating uncertainty.
In the methodology section, report the selection probabilities, base-weight calculation, any subsequent adjustments, and the estimation procedure used.
If weights were trimmed, calibrated, or normalized, describe the rationale and implications. Such changes should not be presented as neutral operations that have no effect on estimation.
Where public-use survey datasets already provide documented final weights and replicate weights or design variables, researchers should follow the survey producer's analytical guidance rather than reconstructing weights from incomplete information.
The goal is not simply to produce weighted numbers. It is to ensure that the estimates and their uncertainty correspond to the population and sampling design the research claims to represent.
