03 · What You Need to Know
Designing a Defensible Sampling Strategy Without a Complete Population List
What Is a Sampling Frame?
A sampling frame identifies the units or selection opportunities through which members of a population can enter a sample.
For example, a university enrollment register may serve as a frame for selecting currently enrolled students.
A sampling frame does not always have to be a single list of individuals. It may consist of geographic areas, institutions, households, establishments, or other units used in a multistage selection process.
The frame should provide an appropriate relationship between the target population and the units available for selection.
Researchers should distinguish the conceptual target population from the operational frame through which members can be reached.
Why Might a Complete Sampling Frame Be Unavailable?
Some populations are not comprehensively registered or are distributed across several independent administrative systems.
For example, freelance academic editors may work independently without belonging to professional associations.
Other populations change rapidly because of migration, employment transitions, enrollment changes, or organizational turnover.
Privacy restrictions may prevent access to existing lists even when those lists are maintained by institutions.
Researchers should determine whether the problem is that no frame exists, an existing frame is incomplete, or a potentially suitable frame cannot be accessed.
These situations may require different solutions.
What Problems Can an Incomplete Frame Create?
An incomplete sampling frame may exclude some members of the target population, include ineligible units, contain duplicates, or provide outdated information.
These problems are often described as coverage errors.
For example, a professional association directory may omit freelance editors who are not members.
If association membership is related to professional experience, income, or AI use, estimates based only on members may differ from those for the intended broader population.
Researchers should examine how sampling frame mismatch affects population inference before selecting a method.
What Sampling Options Are Available Without a Complete Individual List?
| Strategy |
How it works |
Principal limitation |
| Multistage probability sampling |
Selects higher-level units and then eligible individuals within them. |
Requires suitable frames and probability selection at relevant stages. |
| Area probability sampling |
Selects geographic units and identifies eligible members within selected areas. |
May be costly and difficult for geographically dispersed populations. |
| Multiple-frame sampling |
Combines overlapping or complementary selection frames. |
Requires attention to duplicates, overlap, and inclusion probabilities. |
| Time-location sampling |
Selects defined venues and time periods where population members may be encountered. |
Coverage is limited to people attending eligible venues and times. |
| Respondent-driven sampling |
Uses structured network recruitment with specialized estimation assumptions. |
Depends on assumptions about networks and recruitment behavior. |
| Purposive sampling |
Selects participants based on relevant characteristics or experiences. |
Does not ordinarily support design-based population estimation. |
| Conventional snowball sampling |
Uses participant referrals to identify additional people. |
May overrepresent connected networks and omit isolated individuals. |
| Convenience or open recruitment |
Invites accessible or self-selecting participants. |
Unknown selection probabilities and potentially substantial self-selection. |
These approaches have different inferential properties. Researchers should not classify them all as probability sampling merely because some involve random decisions or structured recruitment.
Can Probability Sampling Be Conducted Without a Complete List of Individuals?
Yes. Probability sampling can use several selection stages.
For example, a national education survey may randomly select districts, then schools, and finally students within selected schools.
A complete national list of individual students is not necessarily required at the beginning.
However, appropriate frames and probability selection procedures must exist at the relevant stages, and inclusion probabilities must be determinable.
The design should also address units that fall outside the available clusters or cannot be enumerated.
The absence of one individual-level list does not remove the need for population coverage.
How Does Multistage Sampling Work?
Multistage sampling selects units through successive stages rather than directly selecting all individuals from one complete frame.
For example, researchers studying AI use among schoolteachers might randomly select schools from an appropriate school register and then randomly select teachers within participating schools.
The first stage requires a suitable frame of schools. The second requires an appropriate method for identifying and selecting teachers within those schools.
Researchers should document each stage and calculate or otherwise establish the relevant inclusion probabilities.
Clustering and unequal probabilities may require design-appropriate weighting and variance estimation.
When Is Area Probability Sampling Useful?
Area probability sampling uses geographic units as part of the selection process.
For example, researchers investigating households in a region may randomly select geographic areas and then enumerate or sample households within them.
This approach can be useful when a complete household list is unavailable but suitable geographic frames exist.
However, area sampling may be inefficient for rare or geographically dispersed populations.
It also requires careful attention to boundary definitions, enumeration procedures, and population coverage.
Randomly selecting areas does not automatically establish probability selection of individuals unless subsequent stages are appropriately designed.
Can Several Incomplete Lists Be Combined?
Yes. Multiple-frame sampling may combine complementary sources to improve coverage.
For example, a study of professional researchers might use institutional staff registers, professional association lists, and other appropriate sources.
However, individuals may appear in more than one frame, while others may appear in none.
Researchers must consider overlap, duplication, frame-specific eligibility, and the resulting selection probabilities.
Simply merging lists does not automatically create a complete or probability-based frame.
Where probability selection is intended, the design and estimation procedures should account for the multiple-frame structure.
What Is Time-Location Sampling?
Time-location sampling selects participants through defined combinations of places and time periods where members of a population can be encountered.
For example, a study of workers attending particular professional events may construct a frame of eligible venues and event periods.
Researchers may then select venue-time units and recruit or sample eligible individuals encountered there.
However, people who never attend the covered venues or times have no opportunity to enter through that frame.
Attendance frequency can also influence selection probabilities.
Whether the resulting design supports probability inference depends on the completeness of the venue-time frame, selection procedures, and ability to account for individual attendance patterns.
When Is Respondent-Driven Sampling Considered?
Respondent-driven sampling, or RDS, is a specialized network-based approach developed for certain populations that are difficult to reach through conventional frames.
It generally begins with selected initial participants who recruit additional eligible individuals through structured referral procedures.
Specialized estimators may use information about participants' network connections and recruitment patterns.
However, RDS inference depends on assumptions concerning network structure, recruitment, participation, and other conditions.
These assumptions may be difficult to verify and are not always satisfied.
RDS should not be described as equivalent to conventional probability sampling merely because it uses structured recruitment or statistical weighting.
How Is Respondent-Driven Sampling Different From Snowball Sampling?
Conventional snowball sampling generally asks existing participants to identify or refer additional participants.
The recruitment process may be informal, and population inclusion probabilities are ordinarily unknown.
Respondent-driven sampling uses a more structured recruitment system and specialized estimation methods.
However, structured referrals do not automatically guarantee valid population estimates.
Researchers should distinguish the procedures and assumptions of the methods rather than using the terms interchangeably.
When Is Purposive Sampling Appropriate Without a Frame?
Purposive sampling may be appropriate when the objective is to investigate specific experiences, roles, or conceptual relationships.
For example, a qualitative study of freelance academic editors' experiences with generative AI may deliberately recruit editors with different client arrangements, disciplines, and AI practices.
A complete population list is not necessarily required for such an inquiry.
However, the researcher should justify how participants were identified and why their experiences are relevant.
Purposive sampling does not ordinarily establish the proportion of all freelance editors who hold a particular opinion or use a particular technology.
When Is Convenience Sampling Defensible?
Convenience sampling may be appropriate for preliminary investigations, feasibility studies, or other objectives that do not require design-based population estimates.
For example, researchers may recruit accessible freelance editors to pilot an interview protocol or test whether survey questions are understandable.
However, a convenience sample should not automatically be treated as representative of the broader population.
Its limitations depend on which individuals are accessible and how accessibility relates to the research variables.
Researchers should describe recruitment channels and avoid unsupported population claims.
Can Online Recruitment Replace a Sampling Frame?
Online recruitment can help reach populations distributed across institutions or geographic locations.
For example, researchers may recruit independent professionals through online communities, professional platforms, or public announcements.
However, open invitations ordinarily produce self-selected respondents whose inclusion probabilities are unknown.
Some population members may not use the selected platforms or may be less likely to encounter the invitation.
Using several platforms may improve coverage but does not automatically establish probability sampling.
Researchers should distinguish broad online reach from a defined population selection mechanism.
Can Social Media Advertising Produce a Representative Sample?
Social media advertising may recruit large and diverse groups, but the underlying selection process can be difficult to characterize.
Exposure to advertisements may depend on platform algorithms, user activity, targeting settings, and other factors.
Individuals also decide whether to respond.
Consequently, the probability that each member of the target population enters the sample is generally unknown.
Researchers may use auxiliary information and appropriate modeling to evaluate or adjust some differences, but demographic calibration does not guarantee the elimination of unmeasured selection bias.
How Can Researchers Evaluate Coverage Without a Complete Population List?
Coverage assessment may require combining available information about the population with knowledge of recruitment channels and selection mechanisms.
For example, researchers might examine whether professional association directories omit independent workers, whether online platforms disproportionately reach younger professionals, or whether institutional lists exclude former employees.
Where credible external benchmarks exist, sample characteristics may be compared with them.
However, similarity on measured characteristics does not guarantee similarity on unmeasured variables.
Researchers should explicitly identify population groups that may have no realistic opportunity to enter the sample.
What Is Coverage Error?
Coverage error occurs when the operational selection frame does not adequately correspond to the target population.
Undercoverage occurs when relevant units are missing from the frame.
Overcoverage occurs when the frame includes units outside the intended population.
Duplication can occur when one unit appears multiple times or through several selection routes.
These problems can affect selection probabilities, weighting, and the validity of population estimates.
Researchers should evaluate frame quality before treating the number of accessible records as the size of the target population.
Can Sampling Weights Correct Incomplete Coverage?
Weighting may address some selection differences when relevant information and suitable assumptions are available.
For example, a survey may be calibrated to known population distributions of age, occupation, or geographic region.
However, weighting cannot automatically recover information about groups entirely absent from the sample.
It also cannot guarantee correction for unmeasured characteristics related to the outcome.
Researchers should distinguish design weights based on known inclusion probabilities from model-based adjustments applied to nonprobability samples.
What Is the Problem With Unknown Inclusion Probabilities?
When inclusion probabilities are unknown, researchers cannot directly use conventional design-based estimators that require those probabilities.
For example, an open online survey does not ordinarily establish each target population member's probability of receiving the invitation and responding.
Consequently, conventional probability-sampling margins of error cannot automatically be applied as though the respondents had been selected by simple random sampling.
Model-based estimation may be possible under additional assumptions.
However, those assumptions should be made explicit rather than concealed behind a familiar confidence interval formula.
How Should Sample Size Be Planned Without a Complete Frame?
Sample size planning should follow the research objective and the selection design.
For probability designs, researchers may need to account for clustering, unequal selection probabilities, expected nonresponse, and the precision of intended estimates.
For nonprobability surveys, increasing sample size may improve precision conditional on the observed sample or model assumptions but does not automatically eliminate selection bias.
For qualitative research, sample adequacy may depend on information power, relevant variation, and the analytical strategy.
The absence of a complete frame does not justify choosing an arbitrary sample size without methodological reasoning.
How Should Researchers Handle Duplicate Participants?
Multiple recruitment channels may allow the same individual to appear more than once.
For example, a freelance editor may belong to several professional associations and online communities.
Duplicate records can distort sampling probabilities and sample counts.
Researchers should establish privacy-conscious procedures for identifying duplicate records or responses.
In multiple-frame probability sampling, overlap may need to be incorporated into the design and estimation.
Deduplication procedures should avoid collecting unnecessary identifiable information.
How Can Researchers Reduce Selection Bias in Nonprobability Recruitment?
Researchers may improve coverage by using several relevant recruitment channels, identifying overlooked subgroups, and reducing participation barriers.
For example, a study of independent academic editors might recruit through professional associations, freelance platforms, institutional contacts, and relevant professional communities.
However, adding recruitment channels does not guarantee representative selection.
Researchers should evaluate which individuals remain unlikely to participate and whether those differences may affect the intended findings.
Where possible, sensitivity analyses can examine how conclusions depend on plausible selection assumptions.
What Is a Sensitivity Analysis for Selection Bias?
A sensitivity analysis examines how conclusions might change under alternative assumptions about selection or missing information.
For example, researchers may consider whether an estimated prevalence would differ substantially if nonparticipants were less likely to use generative AI than respondents.
Such analyses can clarify the robustness of conclusions.
However, they do not identify the true population value without additional evidence.
Researchers should distinguish sensitivity scenarios from empirically verified corrections.
When Should the Target Population Be Narrowed?
Narrowing the target population may be appropriate when the available sampling mechanism cannot support the original broader inference and a more focused population remains scientifically meaningful.
For example, a study originally intended to represent all freelance academic editors may be reframed to investigate members of specified professional associations.
However, researchers should not retrospectively redefine the target merely to make the achieved sample appear representative.
Any revision should be substantively justified and transparently reported.
Researchers should distinguish the original target population, the operationally covered population, and the achieved sample.
Can a Study Still Be Valuable Without Population Representativeness?
Yes. A nonprobability sample may provide valuable evidence for qualitative inquiry, feasibility assessment, methodological development, or appropriately qualified analytical questions.
For example, purposively selected freelance editors may provide detailed evidence about how AI tools affect editorial workflows.
However, their experiences do not automatically establish the prevalence of those practices among all freelance editors.
The appropriate conclusion depends on the selection method and intended inference.
Researchers should avoid equating limited population coverage with an absence of scientific value.
How Should Researchers Choose Among Alternative Strategies?
Begin by identifying whether the study requires population estimation or another form of inference.
Determine which frames or selection opportunities exist, including institutional lists, geographic areas, venues, networks, or multiple complementary sources.
Evaluate whether a probability selection mechanism can be established and whether relevant inclusion probabilities can be determined.
If not, select a nonprobability strategy that fits the research objective and describe its assumptions and limitations.
The most defensible approach is not necessarily the one reaching the greatest number of participants, but the one whose selection mechanism best supports the intended claim.
How Should Sampling Without a Complete Frame Be Reported?
Methods sections should describe the target population, available frames or recruitment sources, selection stages, and achieved sample.
For probability designs, report the relevant selection procedures, inclusion probabilities, clustering, and weighting.
For nonprobability approaches, explain how participants were identified, recruited, and selected.
Identify groups likely to be excluded by the selection mechanism and discuss the consequences for inference.
Researchers should not describe a sample as nationally representative or probability-based without evidence supporting those claims.
