03 · What You Need to Know
How to Calculate Survey Invitations and Evaluate Nonresponse
What Is Nonresponse in Research?
Nonresponse occurs when information sought from selected or invited units is not obtained.
It is particularly important in surveys, probability sampling, and studies that depend on voluntary participation.
For example, a university researcher may randomly select instructors from an institutional personnel list, but some selected instructors may never complete the questionnaire.
Nonresponse may result from unsuccessful contact, refusal, limited availability, survey burden, or other circumstances.
Researchers should distinguish nonresponse from ineligibility, because a person who does not belong to the target population should not automatically be counted as an eligible nonrespondent.
Clear disposition records are important for calculating response rates and evaluating recruitment outcomes.
What Is the Difference Between Unit Nonresponse and Item Nonresponse?
Nonresponse can affect entire sampled units or individual questions.
Unit nonresponse
A selected eligible person or unit does not provide the survey response required for inclusion under the survey's response definition.
Item nonresponse
A respondent participates but leaves one or more requested variables unanswered or otherwise unavailable.
For example, an instructor may complete a questionnaire but omit a question about institutional AI policy.
That instructor may still contribute usable information to analyses of other variables.
Consequently, the number of usable observations may differ across outcomes and statistical models.
Researchers should specify whether their recruitment target concerns completed questionnaires, usable primary-outcome responses, or complete records for a particular analysis.
Why Does Nonresponse Affect Sample Size?
A statistical sample size calculation often determines how many usable observations are needed for a specified precision or power objective.
However, not everyone selected or invited will necessarily provide those observations.
If researchers invite only the calculated number, the achieved sample may fall below the target.
For example, inviting 400 eligible instructors with a 60% usable-response rate would yield approximately 240 usable responses on average.
Nonresponse adjustment estimates how many eligible invitations are needed to obtain the planned amount of information.
It does not, by itself, establish that the responding participants adequately represent the target population.
What Is the Formula for Adjusting Sample Size for Nonresponse?
Under a simple recruitment-yield model, the number of eligible invitations required is:
The result should be rounded upward when necessary.
The formula assumes that the expected usable-response proportion is appropriate for the intended recruitment process.
It is a planning approximation, not a guarantee of response or a correction for selection bias.
Why Is Adding the Nonresponse Percentage Incorrect?
Suppose the researcher needs 400 usable responses and anticipates 20% nonresponse.
Adding 20% produces 480 invitations.
With an expected response proportion of 80%, those invitations would yield 480 × 0.80 = 384 usable responses.
Dividing 400 by 0.80 produces 500 invitations, with an expected yield of 400.
The nonresponse percentage applies to the invitation denominator, not the number of responses the researcher wants to obtain.
Using the wrong denominator can systematically underestimate recruitment requirements.
How Does the Expected Response Rate Change the Number of Invitations?
Lower anticipated response rates require more invitations to achieve the same usable sample.
| Expected usable-response rate |
Expected nonresponse |
Eligible invitations for 400 usable responses |
| 90% |
10% |
445 |
| 80% |
20% |
500 |
| 70% |
30% |
572 |
| 60% |
40% |
667 |
| 50% |
50% |
800 |
| 40% |
60% |
1,000 |
All invitation targets are rounded upward.
These values describe expected yield among eligible invitees and do not account for unknown eligibility, incomplete contact information, or additional sampling design requirements.
Actual recruitment may require a more detailed model.
How Is a Survey Response Rate Calculated?
At its simplest, a usable-response proportion can be calculated as the number of usable responses divided by the number of eligible invitations.
However, formal survey response rates may use more detailed definitions.
The American Association for Public Opinion Research provides standardized response-rate formulas that distinguish completed interviews, partial interviews, refusals, noncontacts, known ineligible cases, and cases with unknown eligibility.
Researchers should not label every simple participation fraction as an AAPOR response rate without applying the corresponding definitions.
The calculation method and denominator should be reported transparently.
What If Some Invited People Are Ineligible?
Sampling frames may include records that no longer satisfy the eligibility criteria.
For example, a university personnel list may contain instructors who have left the institution or no longer teach during the relevant semester.
These cases should be distinguished from eligible people who decline or fail to respond.
If the researcher expects that only a proportion e of contacted records will be eligible, the initial contact target may require an additional adjustment.
This is an expected-yield calculation. Formal survey response-rate estimation may require additional treatment of cases with unknown eligibility.
How Should You Estimate the Expected Response Rate?
The expected response rate should reflect the actual recruitment population and procedures.
Relevant information may come from previous institutional surveys, pilot recruitment, comparable published studies, or administrative records.
For example, response to a brief survey distributed through an established faculty communication channel may differ from response to an unsolicited questionnaire sent to unfamiliar external participants.
Survey length, topic sensitivity, invitation credibility, accessibility, and follow-up procedures may also influence participation.
Researchers should evaluate whether historical rates were calculated using comparable eligibility and response definitions.
A response rate borrowed from an unrelated population may provide a misleading recruitment estimate.
Can You Use a Pilot Survey to Estimate Response Rate?
Yes. A pilot can provide information about recruitment yield, contact procedures, questionnaire completion, and eligibility.
However, estimates from small pilots may be uncertain.
For example, obtaining 12 usable responses from 20 eligible invitations produces an observed usable-response proportion of 60%.
That result does not establish that the full survey will achieve exactly 60% response.
The pilot may also involve participants who are easier to reach or more interested than the broader target population.
Researchers should therefore consider plausible response-rate ranges rather than treat a small pilot estimate as exact.
What If Response Rates Differ Across Population Groups?
Different subgroups may respond at different rates.
For example, instructors from departments with established AI policies may be more willing to participate than instructors from departments with limited AI implementation.
These differences can affect both the achieved subgroup sample sizes and the composition of the overall respondent dataset.
When subgroup estimates or comparisons are primary objectives, researchers may need subgroup-specific invitation targets.
However, recruiting additional respondents from underrepresented groups should follow the sampling design and should not be confused with automatically correcting nonresponse bias.
How Do You Calculate Invitations for Separate Subgroups?
Suppose the study requires 100 usable responses from each of two academic sectors.
If the expected usable-response rate is 70% in one sector and 40% in the other, the invitation requirements differ.
The calculation assumes that enough eligible units exist in each subgroup and that the stated response probabilities are plausible.
For probability sampling, unequal selection and response probabilities may require appropriate weighting during population estimation.
Does Nonresponse Affect Statistical Power?
Yes, when nonresponse reduces the number of usable observations available for a planned hypothesis test.
For example, a comparative study requiring 150 usable observations per group may lose power if one group provides substantially fewer responses.
The consequences depend on the achieved group sizes, outcome variability, effect of interest, and planned analysis.
However, power calculations typically address statistical information under assumed models and do not automatically account for nonresponse bias.
Researchers should distinguish reduced precision from systematic differences between respondents and nonrespondents.
See statistical power and study design.
What Is Nonresponse Bias?
Nonresponse bias occurs when the response process creates systematic differences between the estimator based on respondents and the target population quantity.
For example, instructors who regularly use generative AI may be more interested in responding to a survey about AI assessment practices.
If AI users are disproportionately represented among respondents, the observed prevalence of AI use may overestimate the corresponding population proportion.
The amount of bias depends on differences in relevant outcomes between respondents and nonrespondents, not merely on the overall response rate.
Researchers should therefore avoid interpreting a response rate as a direct numerical measure of nonresponse bias.
Does a High Response Rate Guarantee an Unbiased Survey?
No. A high response rate can reduce some concerns about missing information, but it does not establish the absence of nonresponse bias.
For example, even a relatively small proportion of nonrespondents may matter when they differ substantially from respondents on the primary outcome.
Conversely, a survey with a lower response rate may produce reasonably accurate estimates for some outcomes if the response mechanism and adjustment methods support the inference.
Empirical research has shown that response rates alone are imperfect indicators of nonresponse bias.
Researchers should evaluate the characteristics and response patterns relevant to their specific estimates.
Does Reaching the Calculated Sample Size Make the Survey Representative?
No. Numerical sample adequacy and representativeness are separate considerations.
For example, obtaining 400 voluntary responses from an open social media survey does not establish that the respondents represent all university instructors.
Likewise, a probability sample may experience nonresponse that changes the composition of the achieved respondent dataset.
Population inference depends on coverage, selection procedures, response behavior, weighting, measurement quality, and relevant assumptions.
Increasing the number of responses can improve precision under suitable conditions but does not automatically eliminate systematic bias.
Can Nonresponse Weighting Correct the Problem?
Nonresponse weighting may reduce bias when appropriate auxiliary information is available and the assumptions supporting adjustment are reasonable.
For example, a university may know the academic sector, department, and employment category of all sampled instructors.
Researchers may use such information to adjust weights when response probabilities differ across these characteristics.
However, weighting cannot guarantee correction when respondents and nonrespondents differ on important unobserved variables.
Unequal weights may also increase estimator variance.
See how unequal selection probabilities affect sampling weights.
How Can Researchers Evaluate Nonresponse Bias?
Where appropriate information is available, researchers can compare respondents and nonrespondents on characteristics known for the full sampled population.
For example, they may examine differences by academic department, employment status, or institutional sector.
They may also compare weighted respondent distributions with reliable population benchmarks.
However, similarity on observed characteristics does not prove similarity on unobserved outcomes.
Nonresponse bias analysis should focus on variables plausibly related to the research question and response process.
Researchers should report the limitations of available auxiliary information rather than claim that adjustment has eliminated all bias.
Should Researchers Replace Nonrespondents?
Replacing nonrespondents with readily available volunteers may undermine the intended sampling design.
For example, a researcher who randomly selects instructors but replaces nonrespondents with colleagues who are easiest to contact may introduce additional selection bias.
Some probability sampling designs permit prospectively specified supplemental sampling or reserve samples under appropriate procedures.
However, replacement should not be improvised merely to achieve a target number.
See whether replacing nonrespondents compromises probability sampling.
What If Selected Participants Cannot Be Contacted?
Noncontact is one possible source of nonresponse.
Researchers should distinguish failed contact attempts from explicit refusals and confirmed ineligibility.
For example, an instructor may remain eligible but have an outdated email address in the sampling frame.
Documented contact procedures and disposition categories can help clarify recruitment outcomes.
Researchers should use appropriate follow-up methods without violating privacy or consent requirements.
See what to do when selected participants cannot be contacted.
Can Reminder Messages Improve Response Rates?
Reminders may improve participation when they are appropriately timed, accessible, and consistent with the approved recruitment protocol.
For example, an instructor survey may use an initial invitation followed by a limited number of reminders to eligible nonrespondents.
However, the effectiveness of reminders depends on the population, communication channel, survey topic, and other circumstances.
Researchers should avoid assuming that additional reminders will produce a particular response rate.
Excessive contact may also create participant burden or conflict with institutional requirements.
Recruitment planning should combine realistic yield assumptions with respectful contact procedures.
How Does Nonresponse Differ from Attrition?
Nonresponse commonly concerns the failure to obtain data from sampled or invited units during recruitment or survey administration.
Attrition usually refers to the loss of participants or information after enrollment or across follow-up periods.
For example, an instructor who never responds to a survey invitation contributes to unit nonresponse.
A student who enrolls in a longitudinal experiment but misses the final assessment contributes to attrition or missing follow-up data.
The corresponding recruitment adjustments may use similar arithmetic, but the mechanisms and analytical consequences differ.
See how to adjust sample size for expected dropout or attrition.
What If the Sampling Frame Is Too Small to Reach the Target?
A finite sampling frame may not contain enough eligible units to achieve the desired number of responses at the anticipated response rate.
For example, a department with 300 eligible instructors cannot provide 400 distinct instructor responses, regardless of how many invitations are sent.
Researchers may need to invite the entire eligible population, broaden the target population, revise the precision objective, or reconsider the research design.
However, broadening the population changes the scope of inference and should be justified scientifically.
The finite population correction and recruitment yield should be considered separately where applicable.
Researchers should not report an impossible invitation target as though it were achievable.
Does the Response Rate Formula Apply to Convenience Samples?
A recruitment-yield calculation may be useful for convenience samples when the number of people invited and the expected participation proportion are known.
For example, a researcher may estimate how many students must receive an invitation to obtain enough observations for exploratory analysis.
However, a numerical response adjustment does not convert a convenience sample into a probability sample.
Open-link surveys may also lack a clearly defined denominator because the number of eligible people exposed to the invitation is unknown.
Researchers should avoid reporting a conventional response rate when the relevant denominator cannot be established.
How Should Nonresponse Be Reported in the Methodology?
The methodology should identify the required usable sample, anticipated response rate, evidence supporting that assumption, and planned number of invitations.
The sampling frame, eligibility criteria, and recruitment procedures should also be described.
The results should report the number of sampled or invited units, confirmed eligible units, respondents, and analytically usable responses.
Where applicable, use an established response-rate definition such as those provided by AAPOR.
Researchers should describe nonresponse adjustments, weighting procedures, and relevant bias assessments.
Transparent reporting allows readers to distinguish recruitment success from the credibility of the resulting population estimates.
