When Does Adding More Sites Improve Generalizability but Make the Study Harder to Control?

Would Studying More Institutions Make Your Findings More Generalizable or More Difficult to Interpret?

Suppose you are evaluating an AI-supported formative assessment system at one university. You have access to instructors, can monitor intervention delivery, and can ensure that participating classes use the same assessment procedures.

You then consider expanding the study to ten universities. The larger study could include students from different academic programs, institutional environments, and technological conditions. Its findings might become more relevant to decisions beyond the original institution.

But the expansion introduces new questions. Will all universities implement the intervention similarly? Will their assessment instruments produce comparable scores? What happens if some institutions provide extensive instructor training while others offer little support?

A study conducted at more sites may capture a broader range of circumstances, yet those circumstances can also complicate the interpretation of differences in outcomes.

The methodological challenge is determining when site diversity contributes to the intended inference and when it introduces variation that the design cannot adequately measure, manage, or analyze.

Additional Sites Help When Their Diversity Is Relevant and Methodologically Manageable

How Multisite Research Changes Generalizability, Control, and Inference

What Is a Multisite Research Design?

A multisite study collects or analyzes evidence from two or more distinct research settings under a coordinated research question or protocol.

Sites may be universities, schools, hospitals, communities, organizations, geographical locations, or other relevant units.

Multisite research is not a separate quantitative or qualitative methodological approach. It is a structural feature that can be incorporated into experimental, observational, qualitative, or mixed-methods designs.

For example, a multisite randomized trial may evaluate the same intervention across several universities. A qualitative comparative case study may investigate how different institutions implement an educational policy.

The appropriate meaning of a site depends on the question. A school may be a site in one study, while individual classrooms within the same school may function as the relevant clusters in another.

Researchers should distinguish the site of data collection from the unit of intervention assignment, the unit of observation, and the unit of analysis.

Why Might Additional Sites Improve Generalizability?

A single-site study provides evidence under the circumstances represented by that site. Those findings may be informative elsewhere, but their applicability depends on relevant similarities between the original and target settings.

Adding sites can broaden the range of observed populations, institutional arrangements, resources, and implementation conditions.

For example, an AI-supported learning intervention evaluated across universities with different technological infrastructures may provide information about how the intervention performs under a wider range of conditions than a study restricted to one highly resourced institution.

However, external validity concerns the applicability of findings to specified target circumstances. It is not established merely by increasing the geographical spread of participating institutions.

Researchers need to identify the population or settings to which they want to generalize and assess whether the selected sites provide relevant coverage.

Multisite studies can improve the evidentiary basis for external validity when their site selection and analytical strategy reflect that target.

Why Does Site Selection Matter More Than Site Count Alone?

Consider a study conducted at ten universities, all located in major urban areas with extensive technical infrastructure. Its findings may be informative about institutions sharing those characteristics.

However, the study may provide limited evidence about universities with substantially different infrastructure, staffing, or student access conditions.

A smaller study deliberately including contrasting institutional circumstances might provide more useful evidence for investigating contextual variation.

The appropriate selection strategy depends on the intended inference.

If the goal is statistical generalization to a defined population of institutions, probability-based site selection may be relevant, although feasibility and nonparticipation can complicate implementation.

If the goal is to investigate how an intervention operates under contrasting conditions, purposive selection of theoretically relevant sites may be appropriate.

Neither strategy automatically supports every form of generalization. Purposively selected sites may support analytical insights about contextual differences without yielding representative estimates for all institutions.

What Is the Difference Between Internal and External Validity?

A multisite study can improve the relevance of findings to a broader range of settings without necessarily improving internal validity.

For example, adding institutions does not remove confounding in an observational comparison if intervention exposure remains systematically related to student characteristics or institutional resources.

Likewise, a single-site randomized experiment may support a credible causal estimate within that setting while leaving uncertainty about its applicability elsewhere.

Internal and external validity should not be treated as mutually exclusive objectives. A well-designed multisite study can address both, but doing so requires explicit attention to causal identification and the target of generalization.

How Can Site Differences Affect Research Findings?

Participating sites may differ in ways that influence both the intervention and its observed outcomes.

These differences are not automatically methodological defects. Some may be precisely what the study needs to investigate.

The challenge is distinguishing meaningful contextual variation from avoidable inconsistencies in data collection or intervention delivery.

Does Greater Standardization Always Improve a Multisite Study?

Standardization can improve comparability by ensuring that participating sites use common definitions, measurement procedures, eligibility criteria, and data management protocols.

For example, a shared assessment rubric and consistent assessor training may reduce differences caused by scoring procedures rather than student performance.

However, standardizing every aspect of implementation may be inappropriate when the research question concerns how an intervention operates in ordinary institutional settings.

An intervention may require adaptations to fit local curricula, languages, or accessibility requirements.

Researchers should distinguish the intervention's essential functions from features that can vary without changing its intended purpose.

For explanatory trials, tighter delivery control may help isolate a specified intervention contrast. For pragmatic evaluations, allowing some routine variation may improve relevance to actual practice.

The correct degree of standardization follows the research purpose, not a universal rule that every site must operate identically.

When Does Implementation Variation Become a Problem?

Implementation variation becomes problematic when differences in delivery prevent researchers from determining what intervention was evaluated or undermine the intended comparison.

Suppose an AI-supported feedback program requires weekly formative activities. Some universities deliver all activities, while others provide only occasional access to the software.

The sites may then differ not only in context but also in the intervention participants actually receive.

If the research question concerns the effect of offering the program under routine conditions, such variation may be part of the intervention strategy being evaluated.

If the question concerns the effect of a precisely specified instructional procedure, substantial deviations may complicate interpretation.

Researchers should document fidelity, adaptations, exposure, and implementation conditions so that observed variation can be interpreted appropriately.

When implementation is central to the question, evidence about what was actually delivered is essential.

Research on randomized implementation trials similarly emphasizes documenting context because it may influence both implementation processes and the applicability of findings.

How Does Clustering Affect Statistical Analysis?

Participants within the same site may share characteristics or experiences that make their outcomes statistically dependent.

Students attending the same university may encounter similar policies, infrastructure, assessment arrangements, and instructional practices.

Ignoring this dependence can underestimate standard errors and produce misleading confidence intervals or significance tests.

The appropriate analysis depends on the sampling structure, number of sites, assignment mechanism, outcome, and intended estimand.

Possible approaches include multilevel models, generalized estimating equations, cluster-robust variance estimators, and site-level analyses, subject to their respective assumptions and limitations.

However, multilevel modeling does not automatically correct selection bias, confounding, or insufficient site-level information.

Why Does the Number of Sites Matter Independently of the Number of Participants?

Suppose a study includes 2,000 students from four universities. Another study includes the same number of students distributed across 40 universities.

The two studies contain the same number of individual observations but very different amounts of information about variation between universities.

If the question concerns institution-level characteristics or intervention effects assigned at the institutional level, the number of independent sites may be particularly important.

Adding more students within the same few institutions cannot fully compensate for having limited site-level replication.

In cluster-randomized research, observations within clusters are often correlated. This reduces the amount of independent information relative to an equally sized individually randomized study.

Sample-size planning and analysis must therefore consider both the number of clusters and the number of participants within clusters.

How Does Intraclass Correlation Affect the Effective Sample Size?

The intraclass correlation coefficient, commonly abbreviated ICC, describes the degree of similarity among observations within the same cluster under a specified model.

For a simplified design with equal cluster sizes and a common ICC, the design effect can be approximated using the following formula.

The result does not mean that 195 students are discarded or that every analysis has exactly 205 independent observations.

It illustrates how clustering can reduce statistical information. The approximation does not account for all complications, including unequal cluster sizes, covariate adjustment, treatment allocation, or small numbers of clusters.

Researchers should use design-specific sample-size and analysis procedures rather than apply this simplified formula mechanically.

Can Multisite Studies Identify Differences in Intervention Effects?

Multisite research can create opportunities to examine whether intervention effects differ across institutions or contexts.

For example, an AI-supported tutoring program might produce different effects in universities with different instructional arrangements.

Researchers may investigate effect heterogeneity using appropriate interaction analyses, hierarchical models, or other methods.

However, observed differences between site-specific estimates do not automatically establish genuine variation in causal effects.

Site estimates may be imprecise, especially when individual sites have small samples. Differences may also reflect measurement inconsistencies or other biases.

Furthermore, finding that effects differ across sites does not necessarily explain why they differ.

Explanatory claims may require evidence about relevant institutional conditions and an analytical strategy capable of distinguishing competing explanations.

Should Sites Be Treated as Fixed or Random Effects?

Site effects can be represented in different ways, depending on the design and inferential objective.

Fixed site effects can account for measured outcome differences associated with the included sites under a specified model. They are often useful when the analysis concerns the participating sites and site-specific baseline differences.

Random-effects models represent site-level variation through a probability distribution and may support partial pooling across sites.

Neither approach is universally preferable.

The choice depends on the number of sites, sampling process, causal question, model assumptions, and whether the researcher intends to characterize variation beyond the observed institutions.

Importantly, including site effects does not automatically remove confounding from unmeasured participant-level or time-varying factors.

Methodological discussions of multicenter randomized trials emphasize that center-level differences may remain relevant even when a common protocol is used.

What Happens When Sites Use Different Measurement Instruments?

Measurement inconsistency can undermine multisite comparisons.

Suppose one university evaluates writing performance using a 100-point analytic rubric, while another uses a holistic five-category rating.

The resulting scores cannot automatically be pooled as though they represent identical measurements.

Researchers may need common instruments, scoring calibration, linking procedures, or other evidence supporting comparability.

For latent constructs such as trust, motivation, or AI literacy, measurement invariance may be relevant when comparing scores across institutions or language groups.

Using the same questionnaire at every site does not guarantee that respondents interpret its items identically.

Measurement harmonization should therefore be planned before data collection whenever possible.

Can Site Diversity Improve Transferability in Qualitative Research?

Yes, when the sites are selected to investigate meaningful contextual differences or theoretical propositions.

A qualitative multiple-case study might examine how universities with different governance arrangements implement AI assessment policies.

Comparing cases may reveal recurring processes, contrasting interpretations, or conditions under which particular practices emerge.

However, including more cases does not automatically improve the depth or credibility of qualitative analysis.

Researchers need sufficient evidence within each case and a clear rationale for cross-case comparison.

A large number of superficially investigated sites may provide less useful explanatory evidence than a smaller set of carefully selected and deeply examined cases.

The appropriate number of sites depends on the question and analytical strategy rather than a universal threshold.

When Can Adding Sites Make the Study Worse?

Adding sites can weaken a study when the expansion exceeds the research team's capacity to maintain adequate measurement, implementation documentation, recruitment procedures, and analytical quality.

For example, a researcher may recruit many institutions but lack resources to train assessors consistently or verify data quality.

The resulting dataset may contain more participants but also greater measurement error, missing information, and uncertainty about what intervention was delivered.

Additional sites may also introduce logistical delays, inconsistent ethical approval timelines, and differences in data-sharing arrangements.

These challenges do not necessarily make multisite research inappropriate. They mean that the number of sites should be justified in relation to the study's inferential purpose and operational capacity.

More sites are beneficial when they contribute meaningful evidence that can be collected and interpreted reliably.

How Should a Multisite Study Be Planned?

Planning should begin with the target of inference.

If the objective is to estimate an intervention's average effect across a defined population of institutions, the site selection and weighting strategy should reflect that target.

If the objective is to investigate how effects differ across contexts, the design should include meaningful contextual variation and sufficient site-level information.

If the objective is to evaluate a standardized intervention, the protocol should identify which components must remain consistent and which adaptations are permitted.

Researchers should also establish common definitions, measurement procedures, training requirements, data-quality checks, and responsibilities across participating sites.

Statistical planning should reflect clustering, site-level replication, and the intended analysis.

For qualitative multisite research, the plan should address within-case depth, cross-case comparison, and the rationale for selecting each setting.

The study should be expanded only when these requirements can be met without undermining the credibility of its central conclusions.

Expanding an AI-Supported Learning Evaluation From One University to Several

Suppose the intervention produces favorable average writing outcomes across the participating universities, but the estimated effects differ substantially between sites.

The researcher should not immediately attribute those differences to technological infrastructure or instructor training. Other institutional characteristics, measurement differences, and sampling variation may also contribute.

With only eight universities, the study may have limited precision for estimating complex site-level interactions, even if it includes a large number of students.

Nevertheless, the multisite design may provide more informative evidence about implementation under varied institutional conditions than a study restricted to one university.

Its value depends on the credibility of the causal comparison, consistency of measurement, documentation of implementation, and justification for applying the findings to other settings.

Common Misconceptions About Multisite Research

Add Sites Only When They Improve the Evidence Needed for Your Claim

Before expanding a study, determine what the additional sites are expected to contribute.

If the research question concerns an effect within one institution, a carefully controlled single-site design may be sufficient.

If the findings are intended to inform decisions across institutions with different conditions, a multisite design may provide valuable additional evidence.

However, the expansion should be accompanied by a plan for site selection, measurement comparability, implementation documentation, and appropriate analysis.

A useful planning question is whether each additional site increases meaningful evidentiary coverage or merely increases the number of observations.

When the answer is the latter, the expansion may not justify its methodological and logistical costs.

Choose the number and type of sites by working backward from the conclusion the study must support, not from an assumption that larger multisite studies are inherently more persuasive.

Before Expanding to Multiple Sites, Check These Requirements

Frequently Asked Questions About Multisite Research

More Sites Help Only When Their Additional Variation Serves the Research Question

How to Cite This Guide

This guide is intended to be read, shared, and used in research, teaching, and academic work. If you draw on its ideas, explanations, or other content, please acknowledge the source by citing the guide. Doing so gives appropriate credit and helps your readers locate the original resource.