03 · What You Need to Know
Building a Defensible Intervention Design Across Different Analytical Levels
What Is the Unit of Intervention?
The unit of intervention is the entity or level at which an intervention is implemented or delivered.
It may be an individual, classroom, school, hospital, organization, community, or another defined entity.
For example, a school-wide policy may operate at the institutional level even though its effects are experienced by individual students.
Similarly, a hospital-wide infection prevention protocol may change organizational practices while patient-level infection outcomes provide the evidence used to evaluate it.
The intervention unit should be defined according to how the intervention actually operates, rather than simply how the researcher describes it.
Some interventions have several components operating at different levels, which may require a more complex design.
How Is the Intervention Unit Different From the Randomization Unit?
The intervention unit describes where treatment is implemented. The randomization unit describes what is independently assigned to treatment conditions.
These often coincide, but they need not.
For example, a school may be randomized to an intervention that is subsequently delivered through classrooms and teachers.
In that situation, the school is the randomization unit, while classroom-level practices may be important components of intervention delivery.
Alternatively, individual teachers may be randomized to training even though their subsequent instructional practices affect entire classes.
The distinction matters because the assignment mechanism determines the independent randomized units, while the intervention mechanism helps define the causal effect and possible spillovers.
Researchers should distinguish randomization units from measurement units and avoid assuming that every intervention operates at only one level.
What Is the Unit of Outcome Measurement?
The outcome measurement unit is the entity or occurrence for which the outcome is defined and recorded.
For example, a classroom intervention may be evaluated using student achievement, while a hospital intervention may be evaluated using patient complications.
Outcomes may also be measured repeatedly, such as student engagement before and after an instructional program.
These observations provide valuable information about intervention effects, but they may be nested within individuals, classrooms, institutions, or other groups.
The number of outcome observations should not automatically be treated as the number of independently assigned treatment units.
What Are Common Intervention and Outcome Arrangements?
| Intervention level |
Outcome level |
Illustrative research question |
| Classroom |
Student |
Does a classroom instructional strategy improve individual achievement? |
| School |
Teacher |
Does a school-wide professional development policy improve teacher practices? |
| Hospital |
Patient |
Does a hospital safety protocol reduce patient complications? |
| Organization |
Employee |
Does an organizational training program improve employee performance? |
| Community |
Household |
Does a community intervention improve household service access? |
| School |
School and student |
Does an institutional policy change school practices and individual learning outcomes? |
These arrangements can support valid research when assignment, measurement, and analysis are aligned with the intended estimand.
Why Should You Define the Estimand Before Choosing the Design?
An estimand specifies the quantity the research seeks to estimate.
In intervention research, this includes the treatment conditions being compared, the target population, the outcome, and how relevant post-assignment events are handled.
For example, a classroom-level AI tutoring intervention could target the average effect of assigning classrooms to the program on student achievement.
That differs from the effect of actual individual AI tutoring usage, because some students assigned to the intervention may not use the system consistently.
It also differs from the effect of assigning individual students to receive tutoring independently of their classmates.
Researchers should decide whether the principal question concerns assignment, exposure, implementation, or another intervention contrast.
Defining the estimand early helps prevent analytical decisions from changing the scientific question after results are observed.
Should the Intervention Be Randomized at the Group Level?
Group randomization may be appropriate when an intervention is delivered collectively, individual allocation is impractical, or contamination between individual treatment conditions is likely.
For example, a teacher may find it difficult to implement two substantially different instructional approaches independently for students within the same classroom.
Randomizing classrooms can align assignment with actual intervention delivery.
However, cluster randomization may require more groups and participants than individual randomization to achieve comparable precision.
It can also introduce recruitment bias if individuals are enrolled after group allocation becomes known.
The decision should therefore consider intervention feasibility, contamination, ethics, available clusters, and statistical efficiency.
What If Group Randomization Is Not Feasible?
Some interventions are implemented through institutional decisions rather than random assignment.
For example, a university may adopt a digital learning policy while another institution continues with existing practices.
In such cases, treatment selection may be related to institutional resources, leadership, student characteristics, or prior outcomes.
Researchers may consider quasi-experimental designs such as difference-in-differences, interrupted time series, regression discontinuity, or other approaches when their assumptions and data requirements are satisfied.
However, accounting for clustering alone does not eliminate confounding associated with nonrandom treatment assignment.
The design must separately address statistical dependence and causal identification.
How Should Participants Be Recruited Within Intervention Groups?
Recruitment procedures should be planned before treatment assignment whenever feasible.
In cluster-randomized trials, identifying participants after allocation can create selection bias if recruiters or potential participants know the assigned condition.
For example, teachers may encourage particular students to participate after learning that their classroom received an innovative instructional program.
Such differential recruitment can undermine comparability between intervention and comparison groups.
Potential safeguards include identifying eligible participants before randomization, using standardized recruitment procedures, and masking recruiters to allocation where practical.
The implications of recruiting individuals within selected groups should therefore be considered alongside treatment assignment.
How Does Clustering Affect the Analysis?
Individuals within the same intervention group may share treatment exposure and contextual influences.
Students within a classroom share teachers and learning environments, while patients within a hospital may share care protocols.
These common influences can create intracluster correlation.
Ignoring this dependence may produce misleading standard errors or confidence intervals for group-level treatment comparisons.
Researchers should assess the consequences of clustering during research design.
Potential analytical approaches include cluster-level comparisons, mixed-effects models, generalized estimating equations, or suitable cluster-robust inference.
The choice depends on the estimand, outcome distribution, number of groups, and statistical assumptions.
How Do You Calculate Sample Size for Different Intervention and Outcome Units?
Sample size planning should account for the number of independent assignment units, the number of outcome observations within those units, and their expected dependence.
For a simple equal-cluster-size design, a commonly used design-effect approximation is:
Actual sample size requirements depend on the expected effect, outcome variance, allocation ratio, cluster-size variation, baseline adjustment, and planned model.
Researchers should consider whether additional participants or additional clusters provide greater value for the intended comparison.
Why Are Independent Groups Particularly Important?
When treatment is assigned to groups, independent groups provide the assignment replication needed for treatment-effect inference.
Suppose a study includes two intervention schools and two comparison schools, with 200 students measured in each school.
The study contains 800 student outcomes but only four independently assigned schools.
The large student count does not eliminate uncertainty associated with between-school differences.
With very few groups, conventional large-sample methods may be unreliable, and treatment effects may be sensitive to particular institutional characteristics.
Researchers should recruit enough independent groups to support the intended inference rather than relying solely on large within-group samples.
How Should Unequal Group Sizes Be Handled?
Intervention groups may contain different numbers of individuals.
For example, one school may contribute 80 students while another contributes 400.
These differences can affect precision, weighting, and the treatment-effect estimand.
An individual-weighted analysis may target the average effect among individuals, while an equally weighted cluster analysis may target the average effect across groups.
The two quantities can differ when group sizes or treatment effects vary.
Researchers should specify which population quantity matters and choose weighting accordingly.
Unequal group sizes should also be considered during sample size planning rather than treated solely as an analytical inconvenience.
Should Outcomes Be Aggregated to the Intervention Level?
Aggregation can be appropriate when the research question concerns group-level summaries.
For example, researchers may calculate mean student achievement for each classroom and compare classroom means across treatment conditions.
This approach makes the independent assignment units explicit and can simplify interpretation.
However, aggregation may discard individual-level information and may produce a different estimand depending on group weighting.
Individual-level models that account for clustering may preserve more outcome information.
Neither strategy is universally superior. The appropriate choice depends on the number of groups, outcome structure, estimand, and assumptions.
When Are Multilevel Models Appropriate?
Multilevel models can represent individual outcomes nested within intervention groups.
For example, a model of student achievement may include a classroom-level random intercept and individual baseline achievement as a covariate.
This can account for some shared classroom influences while estimating the intervention association.
However, multilevel models rely on assumptions about random effects, outcome distributions, and model specification.
With few groups, estimates and conventional inferential procedures may be unstable or sensitive to those assumptions.
Researchers should select the model according to the intended conditional or marginal estimand rather than assuming that multilevel analysis automatically resolves every design problem.
When Are Generalized Estimating Equations Appropriate?
Generalized estimating equations, or GEE, are often used to estimate marginal or population-averaged associations in correlated outcome data.
They can accommodate continuous, binary, and count outcomes through appropriate link functions.
For example, a GEE model may estimate the marginal effect of classroom-level treatment assignment on individual student achievement while accounting for classroom clustering.
However, conventional robust variance estimation can be unreliable with small numbers of independent clusters.
Small-sample corrections or other suitable methods may be required.
GEE and mixed-effects models may target different parameters, particularly for nonlinear outcomes, and should not be treated as interchangeable.
How Should You Address Treatment Contamination?
Contamination occurs when individuals or groups assigned to one condition receive components of another condition.
For example, students from intervention and comparison classrooms may exchange AI tutoring materials or share learning resources outside scheduled classes.
Teachers may also adopt practices from the intervention in comparison classrooms.
Such contamination can alter the treatment contrast and complicate interpretation.
Researchers should assess whether group randomization sufficiently limits contamination and whether implementation monitoring is feasible.
Contamination should be documented rather than ignored or addressed only after results are observed.
What About Interference Between Individuals?
Interference occurs when one individual's outcome depends partly on the treatment assigned to another individual or group.
In educational research, a classroom intervention may affect peer collaboration, student participation, and teacher behavior.
These within-classroom interactions may be part of the intended intervention mechanism.
However, interactions between intervention and comparison classrooms can create spillovers across assignment units.
Researchers should specify whether the estimand includes within-group spillovers and consider whether between-group interference is plausible.
Randomizing groups may reduce certain forms of contamination but does not guarantee the absence of interference.
What If the Intervention Is Implemented Differently Across Groups?
Group-level interventions may vary in implementation intensity, fidelity, and contextual conditions.
For example, some teachers may integrate an AI tutoring system consistently, while others use it only occasionally.
Implementation variation may influence outcomes and complicate interpretation of the assigned intervention.
Researchers should distinguish the effect of assignment from the effect of actual implementation or usage.
An intention-to-treat analysis in a randomized study generally preserves the original assignment comparison, subject to appropriate outcome and missing-data handling.
Analyses based on actual exposure or adherence may answer different causal questions and require additional assumptions.
How Should Missing Outcomes and Group Dropout Be Handled?
Missingness can occur at the individual or group level.
For example, some students may miss follow-up assessments, while an entire school may withdraw from the study.
Group dropout can be particularly consequential because it reduces the number of independent assignment units.
Individual missingness may also introduce bias when it relates to treatment condition or potential outcomes.
Researchers should specify how missing outcomes will be handled and evaluate assumptions underlying complete-case analysis, imputation, weighting, or other approaches.
Sample size planning should consider plausible attrition at both levels.
How Does the Design Affect Generalizability?
The findings may apply differently to individuals, groups, and institutional settings.
For example, an intervention evaluated in a small number of well-resourced universities may not produce the same effects in institutions with different infrastructure or student populations.
Randomization can support causal inference within the study under appropriate assumptions, but it does not automatically make participating institutions representative of all institutions.
Researchers should distinguish internal validity from the transportability of the intervention effect to other populations or contexts.
Group selection, implementation conditions, and variation in treatment effects should inform the scope of the conclusions.
How Should the Study Be Reported?
Report the intervention unit, randomization or assignment unit, outcome measurement unit, and analytical unit separately.
Describe the number of groups assigned to each condition, the number of individuals measured, group sizes, and recruitment procedures.
Specify the primary estimand, statistical model, clustering assumptions, and handling of missing outcomes.
For cluster-randomized trials, the CONSORT extension provides reporting guidance concerning cluster allocation, participant recruitment, outcome measurement, and analysis.
For nonrandomized intervention studies, researchers should additionally describe the treatment assignment mechanism and the assumptions supporting causal inference.
Transparent reporting allows readers to determine whether the design and analysis support the claimed intervention effect.
