Can Replacing Nonrespondents Compromise Probability Sampling?

Can You Replace Randomly Selected Participants Who Do Not Respond?

You randomly select 500 eligible participants for a survey. After several recruitment attempts, only 350 respond. You still need 150 completed questionnaires to reach the planned sample size.

Would selecting another 150 people solve the problem? What if the replacements are randomly chosen from the same population? Or what if you simply invite the next available participants until the target is reached?

Replacement may seem like a straightforward recruitment decision, but it can change the relationship between the original probability design and the achieved sample. The critical distinction is between obtaining additional observations through a documented probability mechanism and informally substituting people who are easier to recruit.

Unplanned Replacement Can Undermine Probability-Based Inference

Why Nonrespondent Replacement Requires More Than Another Random Draw

What Does Replacing a Nonrespondent Actually Mean?

Replacement occurs when a selected unit does not provide the required data and another unit is recruited in its place. The original unit may have refused, remained unreachable, or failed to complete the study.

Researchers sometimes use replacement to describe several different procedures. These include substituting an available participant, activating a previously selected reserve unit, or drawing an additional probability sample.

These procedures are not methodologically equivalent.

Using the word replacement without explaining the actual procedure prevents readers from evaluating whether the resulting sample supports the intended inference.

Why Does Convenience Replacement Create Selection Bias?

Suppose a researcher randomly selects teachers from a regional registry. When some do not respond, the researcher recruits teachers from nearby schools to restore the desired number of questionnaires.

The replacement teachers are now selected partly because they are accessible and willing to participate. Their probabilities of inclusion are not established by the original probability design.

If accessibility or willingness is associated with the outcome, such as teachers' digital competence or institutional resources, the resulting estimates may be systematically distorted.

Convenience replacement also makes the final selection process difficult to reconstruct. Researchers may no longer know which eligible population members had opportunities to participate or how those opportunities differed.

This problem is not solved merely by ensuring that the replacements share demographic characteristics with the nonrespondents.

Does Randomly Selecting Replacements Solve the Problem?

Not completely. A properly designed supplemental probability sample can retain known selection probabilities for the units drawn. Nevertheless, whether selected individuals respond remains a separate process.

Imagine that teachers with limited internet access are less likely to respond to an online survey. Randomly selecting additional teachers may increase the number of completed questionnaires, but the same response pattern may continue.

The achieved respondents may therefore remain systematically different from the population, even when every invitation originated from a valid random draw.

Researchers should distinguish two questions: whether the additional units were selected probabilistically and whether the combined respondent dataset supports sufficiently unbiased estimation.

The first does not establish the second.

How Do Supplemental Samples Affect Inclusion Probabilities?

When an additional sample is drawn, the overall probability that a population unit enters the combined selected sample depends on how the draws are organized.

For example, suppose two independent probability draws are made from the same frame and units can appear in either draw. The probability that a unit is selected at least once is:

When the second-stage selection depends on observed response outcomes, the unconditional selection probabilities and their estimation may be more complicated. The researcher should not simply assign every supplemental respondent the weight that would apply under an ordinary single-stage draw.

Where the design produces unequal inclusion probabilities, weighting and variance estimation must reflect the implemented mechanism.

What Is a Reserve Sample?

A reserve sample is a set of additional units selected in advance through a defined sampling procedure and held for possible activation under specified conditions.

For example, a researcher might randomly order eligible units and designate an initial release group and subsequent reserve groups before recruitment begins.

This can help maintain operational control over recruitment. However, preselecting reserve units does not automatically guarantee valid population inference.

The protocol must specify when reserve units are activated, whether previously selected units remain eligible for contact, how the overall selection mechanism is represented, and how response outcomes are analyzed.

If reserve activation depends on the number or characteristics of earlier respondents, the final selection mechanism may differ from a fixed-size simple random sample.

A reserve sample is therefore a design feature requiring documentation, not a methodological permission to disregard nonresponse.

Can Researchers Select a Larger Initial Sample Instead?

Often, anticipating nonresponse before data collection is preferable to improvising replacements afterward.

If the researcher requires a specified number of completed questionnaires, the initial number selected can be increased using a defensible anticipated response rate.

This approach makes the response assumption explicit during planning. It also allows researchers to evaluate whether the intended precision remains achievable if the actual response rate differs from expectations.

Why Must Ineligibility Be Distinguished from Nonresponse?

A selected person who does not meet the study's eligibility criteria is not equivalent to an eligible person who refuses or cannot be contacted.

For example, a teacher who left employment before the population reference date may be genuinely ineligible. A teacher who remains employed but does not answer institutional email is an eligible noncontact case.

The distinction affects disposition reporting, response-rate calculations, and the interpretation of frame quality.

Researchers should follow documented eligibility rules and investigate outdated sampling-frame information before classifying selected units.

Confirmed ineligible records may indicate overcoverage in the original frame. Eligible nonrespondents remain relevant to assessing the achieved sample.

What Should Happen Before Any Additional Sampling?

Researchers should first implement the approved recruitment protocol for the originally selected units.

This may involve appropriate reminders, authorized contact-information updates, and permitted alternative contact methods. Refusals and requests to stop contact must be respected.

Procedures for handling unreachable selected participants should be established before recruitment and consistently documented.

Once the achieved response is known, the researcher should examine whether the shortfall primarily affects statistical precision, population coverage, or both.

Additional observations may address insufficient sample size, but they cannot necessarily correct systematic differences between respondents and nonrespondents.

How Should Nonresponse Bias Be Evaluated?

Nonresponse bias depends on the relationship between response behavior and the study outcomes.

A researcher may compare respondents and nonrespondents using characteristics recorded in the original sampling frame, such as geographical region, institutional sector, or academic program.

These comparisons can identify differential response patterns, but similarity on observed characteristics does not establish similarity on unobserved outcomes.

Where appropriate, researchers may use response-propensity adjustments, calibration, or sensitivity analyses. Such methods depend on additional information and assumptions.

Neither a high response rate nor a sufficiently large final respondent count guarantees the absence of nonresponse bias.

Can Supplemental Probability Sampling Be Defensible?

Yes, provided it is implemented as an identifiable probability design rather than an informal replacement procedure.

A defensible supplemental design should specify the eligible frame, selection algorithm, timing, treatment of previously selected units, and the way inclusion probabilities and variance estimation will be handled.

It should also preserve the original recruitment dispositions and explain how the additional sample contributes to the final estimates.

Depending on the design, specialized survey-sampling expertise may be needed to calculate weights and valid standard errors.

The purpose of supplemental sampling should be transparent. Increasing the number of observations and correcting nonresponse bias are different objectives, and the former does not guarantee the latter.

What Should Be Reported When Replacement or Supplemental Sampling Occurs?

The methodology should distinguish the original selected sample, any additional probability samples, and the final responding dataset.

Report the selection procedure for each release or sampling phase, recruitment outcomes, response-rate definitions, and any adjustments made during estimation.

Where appropriate, follow standardized survey disposition reporting such as the American Association for Public Opinion Research's definitions.

Readers should be able to reconstruct how each respondent entered the study and understand the assumptions supporting population-level inference.

Should a Researcher Replace 150 Teachers Who Did Not Respond?

The researcher considers the available responses.

The original selection probability was 500 / 5,000 = 10%, corresponding to a base design weight of 10 for each selected teacher.

However, the 350 respondents cannot automatically be treated as a simple random sample with an inclusion probability of 350 / 5,000. Their participation depended on both the original random selection and their response behavior.

The researcher examines available frame characteristics and discovers that teachers in remote schools responded less frequently than teachers in urban schools.

Because geographical location may be associated with digital competence, recruiting convenient teachers from nearby schools could further distort the achieved sample.

The researcher therefore rejects informal substitution. Instead, the researcher evaluates whether the 350 responses provide adequate precision, whether defensible nonresponse adjustments are possible, and whether a supplemental probability sample is feasible.

If additional probability selection is undertaken, the researcher documents the selection and activation rules, retains the original 150 nonresponse dispositions, and analyzes the combined data using the appropriate design.

The decision is based on preserving inferential credibility rather than merely restoring a numerical target.

Common Misconceptions About Replacing Nonrespondents

How to Decide Whether Additional Sampling Is Defensible

When recruitment falls short, identify the problem before selecting additional participants. Insufficient precision, differential nonresponse, and inaccurate eligibility records require different responses.

In the methodology section, distinguish the initial sample, supplemental selections, and achieved respondents. Explain the reasons for additional sampling and how selection probabilities, response outcomes, and statistical uncertainty were handled.

The defensible objective is not to replace every missing questionnaire. It is to obtain sufficient evidence without misrepresenting how the final observations were selected.

Before Replacing Nonrespondents, Verify the Consequences

Frequently Asked Questions About Nonrespondent Replacement

Additional Recruitment Must Not Conceal the Original Selection Process

How to Cite This Guide

This guide is intended to be read, shared, and used in research, teaching, and academic work. If you draw on its ideas, explanations, or other content, please acknowledge the source by citing the guide. Doing so gives appropriate credit and helps your readers locate the original resource.