Should the Target Population Be Defined Before Choosing Study Sites?

Should You Decide Who Your Research Concerns Before Deciding Where to Collect Data?

Suppose you want to investigate how undergraduate students use generative AI for academic writing. You have colleagues at three universities who are willing to facilitate data collection, so you decide to conduct the study there.

Only afterward do you define your target population as undergraduate students in the Philippines.

But do the three universities adequately represent that population? What if they are all private institutions, located in urban areas, and equipped with relatively advanced digital infrastructure?

The difficulty is not that those institutions are unsuitable research sites. Rather, selecting them before defining the population may allow practical access to determine the evidence available for a much broader research claim.

Should researchers therefore define their target population first, or are there circumstances in which choosing study sites before specifying the population is methodologically defensible?

Usually Define the Intended Population Before Selecting Sites

How Population Definition Should Inform Study Site Selection

Why Does the Order of These Decisions Matter?

Defining the target population establishes the collection of individuals or other entities about which the study intends to make claims.

Study site selection determines the institutions, communities, facilities, or other settings through which data will be collected.

When the population is defined first, researchers can evaluate whether proposed sites cover the settings and characteristics relevant to the research question.

When sites are chosen first, researchers may be tempted to describe the intended population more broadly than the participating settings can support.

For example, findings from students attending three conveniently selected universities do not automatically represent all undergraduate students nationally.

The issue is the relationship between the intended population and the mechanism through which study participants become available.

What Is a Study Site?

A study site is a location, institution, organizational setting, or defined environment in which research activities occur or from which relevant data are obtained.

Examples include schools, hospitals, research laboratories, workplaces, community centers, and geographic areas.

Some studies have one site, while others recruit across multiple settings.

A site may be an administrative location rather than the unit of analysis. For example, a university may serve as the study site while individual students are the analytical units.

Researchers should distinguish the site from the sampling unit, recruitment unit, intervention unit, and unit of analysis.

These entities may coincide, but their roles are not necessarily identical.

How Are Target Population, Accessible Population, and Study Sites Related?

The target population describes the intended population of inference. The accessible population consists of eligible entities that can realistically be reached through available recruitment arrangements.

Study sites often determine important boundaries of accessibility.

For example, if five hospitals agree to participate in a patient outcomes study, eligible patients receiving care in those hospitals may form the accessible population.

However, the target population might include patients receiving care across a wider healthcare system.

Researchers should distinguish the target population, accessible population, and study sample before deciding whether the selected sites provide sufficient coverage.

Access to a site does not establish that the site represents all relevant settings.

What Should Be Decided First?

A defensible planning sequence begins with the research question and intended estimand.

The researcher then identifies the population relevant to that estimand and determines which settings are necessary to observe or recruit its members.

Site selection criteria can subsequently be developed to reflect relevant population characteristics and research requirements.

The sequence is a planning principle rather than a requirement that every decision be finalized independently. Population definitions and site feasibility may be refined iteratively before the protocol is finalized.

What Happens When Sites Are Chosen Only for Convenience?

Convenience-based site selection can restrict population coverage and introduce systematic differences between participating and nonparticipating settings.

For example, universities with established research partnerships may be more willing to participate in educational technology studies.

Those institutions may also differ in infrastructure, institutional policies, academic programs, and student characteristics.

If these differences are related to the outcome or association being studied, the resulting sample may not adequately represent the broader target population.

Convenience selection is not inherently unacceptable. It may be appropriate for exploratory studies, feasibility research, or narrowly defined institutional investigations.

However, researchers should not describe convenient sites as representative without evidence supporting that claim.

Does Randomly Sampling Participants Within Sites Solve Site Selection Bias?

No. Random selection within participating sites does not automatically correct nonrandom selection of the sites themselves.

Suppose three universities are selected through existing professional relationships. Within each university, students are randomly sampled from complete enrollment registers.

The student selection procedure may support probability-based inference within the covered institutional population.

However, students attending universities outside the selected sites have no chance of inclusion through that design.

Consequently, the study does not automatically provide a probability sample of the national undergraduate population.

Researchers should evaluate whether the sampling frame covers the intended population at both institutional and individual levels.

When Is Probability-Based Site Selection Appropriate?

Probability-based site selection is particularly relevant when researchers seek design-based population estimates across a defined set of institutions or geographic settings.

For example, a national school survey might select schools from a comprehensive institutional frame using stratified or multistage probability sampling.

Students could then be selected within sampled schools.

Such designs require attention to selection probabilities, institutional nonresponse, clustering, and appropriate variance estimation.

Probability-based site selection is not necessary for every research objective, but it can strengthen the foundation for population estimation when implemented appropriately.

The sampling design should be chosen according to the intended inference rather than the number of institutions the researcher hopes to include.

When Is Purposive Site Selection Justified?

Purposive site selection can be appropriate when particular settings are selected for their relevance to a phenomenon, theoretical proposition, or comparative question.

For example, researchers might select universities with contrasting AI governance arrangements to investigate how institutional policies influence academic practices.

The purpose would not necessarily be to estimate the national prevalence of a characteristic.

Instead, the selected sites could support detailed examination of policy implementation across meaningfully different contexts.

Researchers should explain why the sites were chosen, what variation they provide, and how the findings may inform understanding beyond those settings.

Purposive selection supports a different inferential logic from probability sampling and should not be presented as though the two were equivalent.

Can a Single-Site Study Be Methodologically Strong?

Yes. A single-site study can be appropriate when the research question concerns a specific institution, setting, intervention, or case.

For example, evaluating the implementation of a new assessment policy within one university may legitimately define that university as the setting of interest.

A single-site study can also support detailed process analysis or provide evidence about mechanisms relevant to broader theoretical questions.

However, the findings should not automatically be generalized statistically to other institutions.

The strength of a single-site study depends on the alignment between its question, design, evidence, and claims, rather than the number of sites alone.

When Is Choosing Sites Before Defining the Population Appropriate?

Site-first planning may be justified when the setting itself defines the research problem.

For example, a researcher might investigate an unexpected institutional policy change at a particular university or examine an intervention introduced by one hospital.

The site is not merely a convenient recruitment location. It is part of the phenomenon being investigated.

Similarly, an intrinsic case study may focus on a particular institution because of its distinctive circumstances.

In such cases, the researcher can define the relevant population in relation to the site while clearly specifying the intended level of inference.

Problems arise when site-first research is subsequently presented as though it had been designed to represent a much broader population.

How Many Study Sites Are Enough?

There is no universal number of sites sufficient for every research design.

For population surveys, the number and distribution of sites should reflect the sampling design, desired precision, clustering, and population structure.

For cluster-randomized interventions, the number of independently randomized sites can be particularly important for statistical power and treatment-effect inference.

For qualitative case studies, site selection may instead emphasize theoretical relevance, contextual variation, and the depth of evidence required.

Researchers should not assume that a study becomes nationally representative simply because it includes several institutions.

Nor should they assume that one institution is inherently insufficient for a question specifically concerning that institution.

How Does Site Selection Affect Statistical Clustering?

Participants recruited within the same institution may share policies, resources, professional practices, or environmental influences.

These shared conditions can create statistical dependence among observations.

For example, students within one university may have more similar experiences of institutional AI policy than students from different universities.

If the analysis ignores relevant clustering, uncertainty estimates may be misleading.

Researchers should consider the implications of clustering in research design when planning site numbers and participant recruitment.

Correctly modeling clustering, however, does not automatically eliminate bias caused by nonrepresentative site selection.

What If Sites Differ Substantially in Size?

Institutions may contribute very different numbers of eligible participants.

For example, one university may enroll 30,000 students while another enrolls 2,000.

These differences affect sampling allocation and potentially the population quantity being estimated.

An analysis targeting the average student may require a different weighting approach from one targeting the average institution.

Researchers should specify whether the intended estimand concerns individuals, institutions, or both.

Unequal site sizes should be considered during sampling and analysis rather than addressed solely through post hoc interpretation.

What If Some Selected Sites Refuse to Participate?

Institutional nonparticipation can alter the intended site sample.

For example, schools with limited staffing or weaker digital infrastructure may be less likely to participate in a demanding technology evaluation.

If these characteristics relate to the study outcomes, site-level nonresponse may affect population inference.

Researchers should document the number of sites approached, eligible, and participating, together with available characteristics of nonparticipating sites.

Replacement procedures should be specified carefully because substituting more accessible institutions may alter selection probabilities or the composition of the study.

When site-level nonresponse cannot be adequately addressed, the resulting limitations should be reported.

Can Selecting Diverse Sites Improve Generalizability?

Selecting sites that differ in meaningful characteristics can improve coverage of relevant contexts and help researchers examine heterogeneity.

For example, including urban and rural universities or public and private institutions may reveal differences in educational technology access.

However, diversity alone does not establish statistical representativeness.

A purposively diverse set of institutions may be valuable for comparative or qualitative inquiry while remaining unsuitable for conventional design-based national prevalence estimation.

Researchers should distinguish contextual diversity from known probability-based population coverage.

The value of site diversity depends on the research objective and the inferential approach.

How Should Site Selection Be Justified in a Research Proposal?

A proposal should explain why the selected sites are appropriate for the target population and research question.

Describe the site selection criteria, institutional characteristics, recruitment arrangements, and any geographic or organizational restrictions.

Where probability sampling is intended, explain the institutional frame and selection probabilities.

Where purposive selection is used, identify the substantive or theoretical rationale for including particular settings.

Researchers should also explain how site selection may affect clustering, population coverage, and the scope of the findings.

The justification should make clear whether sites are intended to represent a wider population or provide evidence about specific contexts.

Choosing Universities for a National Study of AI Literacy

The three-university study could still provide useful evidence about AI literacy among participating students.

However, collecting 1,200 responses would not automatically overcome restricted institutional coverage.

If the research objective genuinely requires national prevalence estimation, the site selection procedure must support that objective.

If the objective instead concerns AI literacy in digitally developed urban university settings, the three institutions may be substantively appropriate, although individual selection and institutional variation would still require attention.

The central decision is whether the study sites were selected to answer the intended question or whether the question was expanded beyond what the selected sites could support.

Common Misconceptions About Choosing Study Sites

How to Decide Whether Population Definition or Site Selection Should Come First

Begin with the scientific question and determine whether the intended inference concerns a broad population or a particular institutional setting.

Then choose a site selection strategy that corresponds to that purpose.

Population-first planning is generally preferable when the study aims to make claims beyond the participating settings. Site-first planning is defensible when the setting is itself central to the research question.

What to Verify Before Finalizing Study Sites

Frequently Asked Questions About Target Populations and Study Sites

Let the Research Question Determine the Population, Then Select Sites That Can Support It

How to Cite This Guide

This guide is intended to be read, shared, and used in research, teaching, and academic work. If you draw on its ideas, explanations, or other content, please acknowledge the source by citing the guide. Doing so gives appropriate credit and helps your readers locate the original resource.