03 · What You Need to Know
Why Heterogeneity Matters in Multisite Research
What Does Heterogeneity Mean in a Multisite Study?
Heterogeneity refers to variation among observations, populations, conditions, or effects. In multisite research, it commonly concerns differences between participating settings.
These differences can arise from participant characteristics, organizational environments, intervention implementation, measurement procedures, or the underlying relationships being investigated.
Importantly, researchers should distinguish variation in baseline outcomes from variation in the effect of an intervention. Two hospitals may have different average patient outcomes while exhibiting similar treatment effects. Conversely, two universities may have similar baseline achievement but respond differently to the same educational intervention.
Between-site outcome variation
Differences in average outcomes or other characteristics across settings, which do not necessarily imply different intervention effects.
Between-site effect heterogeneity
Differences in the estimated relationship or intervention effect across settings, beyond what may be expected from sampling uncertainty.
These concepts require different interpretations and may call for different statistical models.
What Types of Heterogeneity Can Additional Sites Introduce?
Not all site differences have the same methodological consequences.
| Type |
Example |
Possible consequence |
| Population heterogeneity |
Universities recruit students with different prior achievement levels. |
Changes in participant composition and possible effect modification. |
| Contextual heterogeneity |
Hospitals differ in staffing or organizational resources. |
Different baseline risks, implementation conditions, or outcomes. |
| Intervention heterogeneity |
Sites deliver different amounts of training or support. |
Uncertainty about whether the same intervention was evaluated. |
| Measurement heterogeneity |
Sites use different assessment instruments or scoring procedures. |
Reduced comparability and possible measurement bias. |
| Recruitment heterogeneity |
Some sites recruit volunteers while others recruit consecutively. |
Differences in selection mechanisms and participant characteristics. |
| Effect heterogeneity |
An intervention has different effects across institutional environments. |
A pooled effect may conceal meaningful contextual variation. |
The appropriate response depends on whether a difference is an intended feature of the study, an unavoidable contextual condition, or a preventable departure from the protocol.
Why Can a Pooled Result Become Difficult to Interpret?
A pooled analysis combines information across participating sites to estimate a specified quantity. That quantity might be an overall mean, a common treatment effect, or a weighted average of site-specific effects.
When sites differ, the pooled estimate may depend substantially on which institutions contribute the most observations and how the statistical model weights them.
Suppose a digital learning intervention is associated with large improvements at well-resourced universities but little improvement at institutions with limited instructional support. A single pooled estimate could conceal this variation.
The pooled result is not necessarily incorrect. Its usefulness depends on whether the estimated average corresponds to a meaningful target population or policy decision.
Researchers must therefore define the estimand, meaning the precise quantity the study intends to estimate, and explain how the participating sites contribute to it.
Can Heterogeneity Reduce Statistical Power or Precision?
It can, although the consequences depend on the design and analysis.
Greater unexplained variability may increase uncertainty around estimates. In clustered designs, similarity among participants within the same site can also reduce the amount of independent information contributed by a given number of participants.
However, heterogeneity does not invariably reduce statistical power. Adjusting for relevant prognostic factors or site effects may improve precision under suitable conditions, and increasing the number of informative sites may improve estimation of between-site variation.
For cluster-randomized research, the number of clusters, cluster sizes, intracluster correlation, and allocation structure all affect statistical planning.
Researchers should address these features when designing for site effects, rather than relying on an ordinary independent-observations sample-size calculation.
How Can You Tell Whether Differences Are Real or Due to Chance?
Different site-specific estimates do not automatically establish genuine effect heterogeneity. Even if the underlying effect is identical, estimates from separate sites will vary because of sampling uncertainty.
Researchers may examine site-specific estimates and confidence intervals, use interaction terms, or apply models that allow effects to vary across sites.
Where site-level estimates are synthesized using meta-analytic methods, statistics such as Cochran's Q, I-squared, and tau-squared may help describe heterogeneity. Their interpretation depends on the estimation method and available information.
I-squared, for example, is commonly used to describe the proportion of observed variability in effect estimates attributable to heterogeneity rather than sampling error within a meta-analytic framework. It is not the percentage of sites that disagree, nor a direct measure of the magnitude of contextual differences.
Tests for heterogeneity may have limited power when few sites are available. A nonsignificant test should not be treated as proof that effects are identical.
When Is Heterogeneity Scientifically Valuable?
Heterogeneity can be informative when the research question concerns applicability across settings or the conditions under which an intervention succeeds.
For example, differences in intervention effects across schools may suggest that teacher preparation, connectivity, or implementation support warrants further investigation.
Such variation may help researchers refine theories, identify implementation requirements, and develop hypotheses for future studies.
Nevertheless, exploratory explanations should remain appropriately tentative. An observed association between a site characteristic and an intervention effect does not establish causation, particularly when only a few sites are available.
The benefits of including multiple sites are greatest when relevant variation is anticipated and the study collects enough information to interpret it.
When Does Heterogeneity Threaten Comparability?
Some differences are scientifically meaningful; others undermine the validity of combining observations.
If sites apply different definitions of eligibility, use instruments that measure different constructs, or implement substantially different interventions, the resulting data may not support the originally intended pooled comparison.
Measurement inconsistency is especially consequential. A score of 80 on one institution's examination cannot automatically be treated as equivalent to a score of 80 on another examination.
Appropriate measurement standardization across sites can reduce avoidable variation, while carefully justified adaptations may be needed where identical procedures would not produce equivalent measurements.
Likewise, differences in recruitment procedures can create selection differences that complicate interpretation.
Should Researchers Use Fixed or Random Site Effects?
Fixed-effects and random-effects approaches offer different ways of representing site-related variation. Neither is universally superior.
A model with fixed site indicators can account for differences in baseline outcome levels between the included sites, subject to the model's assumptions. It does not automatically address differences in treatment effects or all sources of confounding.
A random-intercept model represents site-level baseline variation through a probability distribution. Additional random slopes or interactions may be needed when effects themselves are expected to differ.
The choice depends on the estimand, sampling of sites, number of sites, assignment mechanism, model assumptions, and inferential objective.
With very few sites, estimating variance components or using conventional cluster-robust standard errors may be unreliable without suitable methods or corrections.
Researchers should select the approach during design and analysis planning, with specialist statistical input when necessary.
Should Site-Specific Results Always Be Reported?
Site-specific results can improve transparency, especially when contextual variation is central to the research question.
However, site-level estimates may be imprecise when individual sites contribute small samples. Presenting numerous estimates without uncertainty intervals can invite overinterpretation.
Reporting must also consider confidentiality, particularly where small institutional units or sensitive populations could be identified.
When feasible, researchers should describe relevant site characteristics, recruitment patterns, protocol deviations, and the distribution of observations across sites.
The objective is not to generate a separate headline conclusion for every institution, but to provide sufficient evidence for readers to understand the pooled findings and their limitations.
Watch Out
Do not select a pooled or site-specific result merely because it is statistically significant or more favorable. Decisions about pooling, subgroup comparisons, and site-by-treatment interactions should be prespecified where possible. Exploratory analyses should be identified transparently.
04 · A Practical Example
When a Positive Overall Result Conceals Different Site Experiences
Hypothetical Example
An AI-Assisted Writing Intervention Across Four Universities
A researcher evaluates an AI-assisted formative feedback intervention at four universities. For illustration, assume each university contributes an equally weighted estimate of the intervention's effect on a common writing assessment, expressed as the difference in mean scores between intervention and comparison groups.
| University |
Estimated score difference |
Contextual characteristic |
| A |
+12 points |
Extensive instructor training |
| B |
+8 points |
Moderate instructional support |
| C |
+2 points |
Limited implementation support |
| D |
-2 points |
Frequent connectivity interruptions |
This arithmetic average is not necessarily the appropriate pooled treatment-effect estimator. A defensible analysis would require information about sample sizes, allocation, variance, clustering, and the intended estimand.
Nor can the apparent association between institutional support and estimated effects establish that support caused the differences. Sampling uncertainty, participant characteristics, and other site conditions could contribute.
The practical implication is that the researcher should examine the distribution and uncertainty of site-specific estimates, assess protocol implementation, and interpret the overall result in relation to the intended target settings.
