03 · What You Need to Know
What Multisite Research Can and Cannot Establish About Generalizability
What Is Multisite Research?
Multisite research collects or analyzes evidence from more than one research location, institution, organization, community, or other defined setting.
In educational research, sites may include universities, schools, campuses, departments, or learning centers. In healthcare research, they may include hospitals, clinics, or service systems.
Multisite designs vary substantially. Some studies recruit participants independently at several institutions using a common protocol. Others randomize institutions to interventions, compare naturally occurring institutional practices, or investigate how implementation differs across contexts.
The site is not necessarily the unit of analysis or randomization. A study may recruit students from several universities while randomizing students individually within each institution.
Alternatively, a cluster-randomized trial may assign entire universities or schools to intervention conditions.
These distinctions matter because the design determines what comparisons are credible and how dependence among observations should be handled.
Why Do Researchers Expect Multiple Sites to Improve Generalizability?
A single-site study observes participants within one institutional or environmental context. Its findings may depend partly on characteristics of that setting.
For example, the effectiveness of an educational technology intervention may be influenced by connectivity, instructional support, class size, assessment practices, or institutional policy.
Adding sites can provide evidence about whether the intervention operates similarly under different conditions.
It may also reduce reliance on the assumption that one institution adequately represents every setting of interest.
However, this benefit depends on the characteristics of the additional sites. Five nearly identical institutions may provide less information about contextual variation than a smaller number selected to represent substantively different environments.
The relevant issue is not simply geographical expansion. It is whether the study observes variation that matters for the research question.
What Is the Difference Between More Sites and More Contextual Diversity?
The number of sites describes how many distinct settings participate. Contextual diversity concerns how those settings differ in characteristics relevant to the findings.
Number of research sites
The count of institutions, locations, or other settings contributing observations to the study.
Contextual diversity
The variation across sites in characteristics that may influence the outcome, intervention delivery, or treatment effect.
Suppose ten universities have similar technological infrastructure, student demographics, and instructional practices. The study may provide substantial evidence about institutions with those characteristics.
However, it may provide little direct evidence about institutions operating under substantially different conditions.
By contrast, a study involving universities with different resource levels and instructional arrangements may provide more information about the relationship between context and intervention performance.
Neither approach is universally superior. The appropriate site composition depends on the population to which the findings are intended to apply.
Does Multisite Research Make the Sample Representative?
No. Recruiting participants from multiple institutions does not automatically create a probability sample of the intended population.
If institutions volunteer to participate, they may differ systematically from institutions that decline or are never approached.
For example, universities with established research offices and strong digital infrastructure may be more willing to participate in educational technology evaluations.
Students recruited within those universities may also differ from the broader student population.
Consequently, selection can occur at both the institutional and individual levels.
A probability-based sampling design may support population inference when inclusion probabilities, coverage, response processes, and estimation methods are appropriate.
Purposive or convenience-based site selection can also produce useful evidence, but broad statistical generalization requires additional assumptions.
The distinction between sample resemblance and defensible population representation remains important in multisite research.
Why Does Site Selection Matter More Than Site Count Alone?
Site selection determines which institutional conditions are observed and which remain outside the evidence.
Suppose a research team evaluates an intervention intended for universities throughout a national higher education system.
If the team recruits only institutions with advanced learning management systems, it may exclude settings where implementation challenges are most consequential.
Adding more technologically advanced universities may increase precision for the participating institutional context without substantially improving evidence about resource-constrained institutions.
Researchers should therefore identify site characteristics that could influence outcomes or modify intervention effects.
Relevant characteristics might include institutional resources, geographical context, student composition, instructional arrangements, and technical support.
These should be chosen because of their plausible relationship to the research question, not merely because they provide convenient categories for reporting.
When the objective is to examine variation across settings, selecting sites for relevant diversity rather than convenience alone may be more informative.
What Is Treatment-Effect Heterogeneity Across Sites?
Treatment-effect heterogeneity occurs when an intervention's causal effect differs across participants, populations, or settings.
For example, a digital learning intervention may produce a larger effect at institutions with extensive instructional support than at institutions where students receive little technical assistance.
Alternatively, the intervention may benefit students at less-resourced institutions more because it provides learning opportunities that were previously unavailable.
Neither direction should be assumed without evidence.
Multisite studies can investigate whether estimated effects differ across institutional contexts, provided the design and analysis support such comparisons.
However, observed differences between site-specific estimates may reflect sampling variation, measurement differences, confounding, or implementation differences rather than genuine causal effect modification.
Researchers should distinguish evidence of effect heterogeneity from unexplained variation in observed outcomes.
Does Variation in Average Outcomes Mean Treatment Effects Vary?
No. Sites may have different average outcomes even when the intervention has a similar causal effect across them.
For example, students at one university may achieve higher examination scores because of stronger prior preparation, regardless of intervention assignment.
That difference in baseline achievement does not necessarily mean the intervention effect is larger at that university.
Conversely, sites with similar average outcomes may experience different intervention effects.
Researchers should distinguish site-level prognostic characteristics from factors that modify the treatment effect.
This distinction is particularly important when the purpose of multisite research is to determine whether a causal finding applies across settings.
How Does Multisite Research Affect Internal Validity?
Adding sites does not automatically strengthen internal validity.
Each site's assignment procedures, intervention delivery, outcome measurement, and follow-up processes must support the intended causal comparison.
A multisite randomized trial may provide credible within-site comparisons when randomization is properly implemented.
However, site-specific measurement differences, selective recruitment, differential attrition, or deviations from intended interventions may still introduce bias.
In nonrandomized studies, treatment assignment may be confounded with site characteristics.
For example, if all intervention participants attend one group of universities and all comparison participants attend another, the treatment effect may be difficult to separate from institutional differences.
Researchers should therefore evaluate internal validity independently from the number of participating sites.
Why Must Multisite Analyses Account for Clustering?
Participants within the same institution may have more similar outcomes than participants from different institutions.
They may share instructors, resources, institutional policies, or other contextual influences.
This dependence is often described using the intraclass correlation coefficient, or ICC.
When outcomes are correlated within sites, treating every participant as statistically independent may underestimate uncertainty.
Appropriate analytical methods may include multilevel models, cluster-robust standard errors, or design-based procedures, depending on the assignment mechanism, number of clusters, and estimand.
These methods are not interchangeable in every situation. Cluster-robust procedures, for example, may require special care when the number of clusters is small.
The statistical analysis should reflect the actual data structure rather than treating site membership as an incidental descriptive variable.
Can Multilevel Models Establish Generalizability?
Multilevel models can represent variation among participants and sites and may estimate site-specific or population-level relationships under specified assumptions.
They are useful when data have a hierarchical structure, such as students nested within universities.
However, fitting a multilevel model does not automatically make the participating institutions representative of a broader population.
The interpretation of site-level random effects depends on assumptions about the distribution from which sites arise.
If institutions were selected through a highly selective convenience process, treating them as though they were randomly sampled from all universities may be difficult to justify.
Researchers should distinguish statistical modeling of site variation from evidence supporting inference to unobserved institutions.
Does Adding More Sites Always Increase Statistical Power?
Not necessarily. Statistical power depends on the number of sites, participants per site, outcome variability, within-site correlation, assignment design, and expected effect.
In cluster-randomized studies, increasing the number of independent clusters may be particularly valuable because participants within the same cluster provide partly redundant information.
However, adding sites with very small samples, inconsistent measurement, or substantially different intervention implementations may create other analytical challenges.
Increasing the number of participants within existing clusters may also produce diminishing gains in precision when within-cluster correlation is substantial.
Researchers should conduct design-specific sample-size calculations rather than assuming that more sites or more participants always provide equivalent benefits.
The question of how many sites are needed to capture meaningful contextual variation is related to, but distinct from, statistical power for the primary treatment effect.
Can a Multisite Study Be Too Heterogeneous?
Heterogeneity is not inherently a methodological defect. It may be precisely what researchers need to investigate.
However, combining settings with substantially different interventions, outcome definitions, or measurement procedures can make the pooled estimate difficult to interpret.
For example, one university may implement a complete adaptive learning program while another provides only occasional access to selected materials.
If the interventions differ substantially, a pooled estimate may represent an average across different treatment strategies rather than a single well-defined intervention.
Researchers should specify which implementation differences are acceptable variations within the intervention and which constitute substantively different treatments.
They should also establish consistent outcome definitions or appropriate measurement linking where necessary.
What Does a Pooled Multisite Effect Actually Represent?
A pooled effect summarizes information across participating sites according to a specified estimand and weighting structure.
It does not automatically represent the average effect across all institutions in a broader population.
For example, an individual-level analysis may give greater influence to universities contributing more students.
An equally weighted site-level analysis may instead give each institution similar influence regardless of enrollment.
These approaches can estimate different quantities when treatment effects vary across sites.
Researchers should determine whether the target is an average effect among participating students, an average across participating institutions, or an effect in a broader target population.
The appropriate weighting and analysis follow from that decision.
How Can Researchers Evaluate Generalizability Beyond Participating Sites?
First, define the population of institutions or individuals to which the findings are intended to apply.
Then compare the participating sites with that population on characteristics plausibly related to the outcome or treatment effect.
Researchers may use institutional statistics, administrative records, population benchmarks, or other relevant data sources.
If important contextual effect modifiers are measured and sufficiently represented, formal generalization or transportability methods may be appropriate under defensible assumptions.
However, statistical adjustment cannot automatically recover treatment effects for institutional conditions absent from the study.
The distinction between generalization and transportability across populations helps clarify the assumptions involved.
What Should Researchers Report About Their Sites?
Researchers should describe how sites were identified, approached, selected, and enrolled.
They should report relevant institutional characteristics, participant recruitment procedures, intervention implementation, outcome measurement, and the distribution of participants across sites.
Where appropriate, explain whether site-level variation was anticipated and how it was addressed analytically.
For cluster-randomized trials, the CONSORT extension for cluster trials provides reporting guidance concerning cluster recruitment, allocation, participant flow, and analysis.
Detailed contextual reporting helps readers assess whether findings are applicable to settings beyond those observed.
Simply reporting that a study was conducted at several institutions is insufficient to establish what those institutions represent.
