03 · What You Need to Know
How to Select Research Sites That Contribute Meaningful Evidence
Why Does Research Site Selection Matter?
Research sites are not merely locations where investigators collect data. They are environments in which participants encounter institutional practices, resources, policies, and social relationships that may influence the phenomenon being studied.
For example, an educational technology intervention may operate differently depending on internet connectivity, instructor preparation, class size, technical support, and students' prior digital experience.
If all participating institutions share similar characteristics, the study may provide strong evidence about those conditions while offering limited information about substantially different environments.
Site selection therefore influences which contextual processes researchers can observe and which remain outside the study.
It may also influence the population represented by the findings, although the relationship between site selection and generalizability depends on the research design.
What Is Convenience-Based Site Selection?
Convenience-based site selection recruits settings primarily because they are accessible, willing to participate, affordable to reach, or administratively straightforward.
For example, a researcher may recruit universities with which they already have professional relationships.
This approach can reduce costs, facilitate permissions, and make complex research feasible.
However, accessible institutions may differ systematically from those that are difficult to recruit.
Universities with strong research cultures, established digital infrastructure, or experienced administrative teams may be more willing to participate in educational technology evaluations.
If these characteristics influence intervention implementation or outcomes, the participating sites may provide an incomplete picture of the intended application.
Convenience sampling is therefore not automatically inappropriate, but its inferential limitations should be recognized.
What Is Diversity-Oriented Site Selection?
Diversity-oriented site selection deliberately includes settings that differ along characteristics relevant to the research question.
In qualitative methodology, maximum variation sampling is one approach used to examine a phenomenon across contrasting conditions.
For example, researchers might select universities with high, moderate, and limited digital infrastructure to investigate how an instructional intervention is implemented under different resource conditions.
The objective is not necessarily to reproduce the numerical distribution of institutions in the broader population.
Instead, the selection aims to make important contextual contrasts observable.
Such designs can help researchers identify common patterns, context-dependent mechanisms, and implementation challenges that might remain invisible in a homogeneous set of sites.
However, deliberately including contrasting cases does not establish how frequently those conditions occur across the target population.
How Is Contextual Diversity Different From Statistical Representativeness?
Contextual diversity and statistical representativeness address different inferential objectives.
Contextual diversity
Deliberately including settings with meaningful differences to investigate how a phenomenon operates across conditions.
Statistical representativeness
Supporting population inference through an appropriate selection and estimation strategy, often involving probability sampling.
Suppose researchers select one highly resourced university, one moderately resourced university, and one institution with limited technological infrastructure.
These sites may provide informative contextual contrasts.
However, the three institutions do not automatically represent the national distribution of universities or students.
Conversely, a probability sample of institutions may support population estimation even if its realized sample does not maximize contextual contrast.
Researchers should therefore distinguish the requirements for sample representativeness from the logic of purposive contextual selection.
When Is Convenience Sampling Defensible?
Convenience-based site selection may be appropriate when the research question is explicitly limited to accessible institutions or when the study is exploratory, developmental, or focused on feasibility.
For example, a pilot study may investigate whether an intervention can be implemented reliably at one cooperating university before a broader evaluation is attempted.
The purpose may be to identify technical problems, refine procedures, or examine the acceptability of an intervention.
Such findings can be useful without claiming that the participating institution represents all universities.
Convenience sampling may also be unavoidable when access is restricted or resources are limited.
The methodological concern arises when researchers use convenience-selected sites but make conclusions requiring evidence from a broader or different population.
When Should Contextual Diversity Be Prioritized?
Contextual diversity becomes particularly important when the research question concerns variation across settings or when contextual characteristics plausibly influence the outcome or intervention effect.
For example, researchers evaluating an online learning intervention may want to understand whether limited connectivity changes students' ability to participate.
Recruiting only institutions with reliable connectivity would provide little direct evidence about that question.
Similarly, a study investigating implementation barriers across higher education institutions may need settings with different administrative arrangements and resource constraints.
Researchers should identify the contextual dimensions that matter before selecting institutions.
The objective is not diversity for its own sake. It is diversity that provides evidence about theoretically or practically relevant differences.
How Do You Identify Relevant Contextual Characteristics?
Begin with a substantive explanation of how institutional conditions could influence the phenomenon being studied.
For educational technology research, relevant characteristics might include connectivity, access to devices, faculty preparation, institutional support, and the organization of instruction.
Researchers can use prior literature, preliminary interviews, administrative data, and implementation experience to identify plausible contextual mechanisms.
However, characteristics should not be selected merely because they are easy to classify.
For example, geographical location may correlate with infrastructure differences, but location alone does not explain how an intervention operates.
Researchers should identify the underlying conditions that make geographical variation relevant.
Where possible, collect direct information about those conditions rather than relying exclusively on broad institutional labels.
Can Too Much Diversity Make a Study Difficult to Interpret?
Yes, particularly when sites differ in several consequential characteristics simultaneously.
Suppose a study includes institutions that differ in curriculum, student preparation, assessment systems, technological resources, and intervention delivery.
If outcomes vary across sites, researchers may struggle to determine which differences explain the observed patterns.
This does not mean that heterogeneous settings should be avoided.
It means the design should distinguish contextual variation relevant to the research question from variation that makes comparisons difficult to interpret.
Researchers may need common outcome definitions, carefully documented implementation procedures, or analytical strategies that account for important site-level characteristics.
In qualitative research, detailed within-case analysis may help preserve contextual meaning before comparing patterns across sites.
Should Researchers Select Extreme or Typical Cases?
Extreme cases may reveal mechanisms or implementation challenges that are difficult to observe in ordinary settings.
For example, a university with severe connectivity limitations may provide valuable evidence about barriers to implementing an online learning platform.
However, extreme cases may not reflect the conditions encountered by most intended users.
Typical cases may be useful for examining ordinary implementation, but defining what counts as typical requires information about the relevant population.
Researchers may combine contrasting, typical, and strategically important cases when the research purpose justifies doing so.
The selection logic should be explicit rather than presented as though every purposively selected site were representative.
Can Site Selection Affect Internal Validity?
Yes, depending on how sites enter the causal comparison.
Suppose intervention participants are recruited from well-resourced universities while comparison participants come from institutions with limited resources.
Institutional differences may then be confounded with intervention assignment.
The observed outcome difference could reflect the intervention, institutional characteristics, or both.
Randomization within sites or other credible assignment strategies may help separate treatment effects from site differences.
However, site selection and treatment assignment are distinct processes. A study may have strong internal validity among participating institutions while still facing limitations in generalizing beyond them.
Researchers should therefore assess the causal comparison separately from the broader applicability of the participating sites.
Does Selecting Diverse Sites Automatically Improve External Validity?
No. Diversity can provide information about how findings vary across observed contexts, but it does not automatically establish population representation.
For example, including one institution from each resource category may reveal important differences, but it does not establish the prevalence of those categories or the average treatment effect across all institutions.
Researchers may need population data, probability-based selection, or defensible transportability assumptions to support broader quantitative generalization.
Furthermore, diversity is useful only when the selected dimensions are relevant to the intended inference.
The broader question is whether the multisite design provides evidence that supports generalization, not simply whether the institutions differ.
How Many Diverse Sites Are Needed?
There is no universal number.
The appropriate number depends on the research question, contextual dimensions, analytic approach, and precision or depth required.
A small comparative case study may use a limited number of deliberately contrasting institutions to investigate implementation processes.
A quantitative study estimating site-level treatment-effect heterogeneity may require substantially more independent settings.
Researchers should not treat the number of contextual categories as equivalent to the number of sites required for statistical inference.
The question of how many sites are needed to capture contextual variation should be answered through design-specific reasoning rather than a fixed numerical rule.
Can Researchers Combine Convenience and Diversity?
Yes. Practical accessibility and contextual relevance need not be treated as mutually exclusive.
Researchers may identify theoretically important categories of institutions and then recruit feasible sites within each category.
For example, a research team might seek universities with high, moderate, and limited technological infrastructure while considering administrative access and implementation costs.
This approach may improve contextual coverage relative to unrestricted convenience sampling.
However, recruiting accessible institutions within categories does not automatically create probability sampling.
The study should describe both the substantive selection criteria and the practical constraints that influenced recruitment.
How Can Researchers Avoid Selecting Sites Based on Expected Results?
Site selection can become problematic when researchers deliberately choose institutions expected to produce favorable findings without acknowledging that selection.
For example, selecting only universities with enthusiastic instructors and strong technical support may make an intervention appear more feasible than it would be under ordinary conditions.
These institutions may still be appropriate for an explanatory study, but the selection should match the research purpose.
Researchers should establish site-selection criteria before examining intervention outcomes whenever possible.
They should document reasons for inclusion, exclusion, refusal, and replacement.
Transparent procedures reduce opportunities for outcome-driven selection and allow readers to evaluate the resulting evidence.
How Should Site Selection Be Reported?
Research reports should describe the target population, site-selection strategy, contextual criteria, recruitment process, and relevant characteristics of participating institutions.
Where feasible, report how many sites were approached, how many participated, and why others declined or were excluded.
Researchers should explain whether sites were selected for statistical representation, theoretical contrast, feasibility, or another purpose.
They should also identify important contextual conditions absent from the study.
This information helps readers assess whether the findings apply to different settings.
Simply describing sites as diverse or representative is insufficient without explaining the basis for those descriptions.
