03 · What You Need to Know
How to Turn Individual Observations Into Defensible Group-Level Evidence
What Makes a Research Question Group-Level?
A research question is group-level when its principal outcome, relationship, or comparison concerns groups rather than differences among individual members.
For example, asking whether students with stronger engagement achieve higher examination scores is an individual-level question. Asking whether classrooms with higher average engagement have higher average achievement is a classroom-level question.
Although the same student records may contribute to both analyses, the estimands differ.
In the first case, the researcher examines variation among students. In the second, the researcher examines variation between classrooms.
Determining the unit of analysis before aggregation helps ensure that the analytical method answers the intended question.
How Can Individuals Provide Information About a Group?
Individuals can contribute measurements of group characteristics in several ways.
They may report their experiences within a shared environment, provide assessments of organizational practices, or supply outcomes that can be summarized at the group level.
For example, students can report classroom instructional support. Employees can describe organizational communication. Patients can contribute outcome records used to estimate hospital-level event rates.
However, the meaning of the resulting aggregate depends on what was measured.
An average of students' personal engagement scores represents average individual engagement within a classroom. It does not automatically measure a shared classroom engagement climate.
Similarly, the proportion of employees reporting dissatisfaction describes the distribution of individual dissatisfaction. Calling it organizational dysfunction requires additional conceptual justification.
What Is Aggregation?
Aggregation combines observations from lower-level units into a summary defined at a higher level.
Common summaries include group means, proportions, totals, rates, medians, and measures of dispersion.
The appropriate summary depends on the construct and research question.
| Individual information |
Possible group measure |
Group-level interpretation |
| Student examination scores |
Classroom mean score |
Average achievement among represented students in a classroom |
| Employee survey responses |
Organization mean or response distribution |
Average perception or distribution of perceptions within an organization |
| Patient readmission outcomes |
Hospital readmission proportion or rate |
Recorded readmission frequency among the relevant hospital population |
| Household internet access |
Community proportion with access |
Connectivity coverage among represented households |
| Individual research outputs |
Institutional publication total |
Output attributed to the institution under specified counting rules |
Different aggregation rules answer different questions. A hospital's total number of readmissions, for example, is strongly influenced by patient volume. A readmission rate incorporates a denominator but may still require attention to case mix and outcome ascertainment.
How Do You Know Whether Aggregation Is Conceptually Justified?
Begin with the construct. Is it inherently a group property, a shared perception, or merely a summary of individual characteristics?
A classroom's student count is a group property. Average student achievement is a compositional summary. Classroom climate may be conceptualized as a shared characteristic, but its measurement requires evidence that the items and responses adequately capture that collective construct.
These distinctions affect interpretation.
For instance, a classroom with high average engagement and substantial disagreement among students differs from a classroom in which nearly everyone reports similar engagement, even if their means are identical.
Researchers should specify whether they intend to estimate an average, characterize consensus, or measure a contextual property.
The theoretical justification for aggregation should precede the statistical calculation.
When Does Within-Group Agreement Matter?
Within-group agreement concerns how similarly members of the same group respond to a measure.
It is particularly relevant when the construct is intended to represent a shared perception, such as organizational climate or classroom instructional environment.
If respondents within a group disagree substantially, describing their responses as one shared perception may be difficult to defend.
Agreement statistics such as rWG may be considered for certain survey-based constructs, provided the reference distribution and interpretation are justified.
However, high agreement is not necessary for every group-level summary. A classroom's mean examination score remains a meaningful descriptive average even when students' scores vary widely.
The need for agreement evidence therefore depends on whether the researcher claims consensus or simply summarizes a distribution.
What Is the Intraclass Correlation Coefficient?
The intraclass correlation coefficient, or ICC, can quantify the proportion of variance associated with differences between groups under a specified statistical model.
For a simple random-intercept model, ICC(1) is commonly expressed as the between-group variance divided by the sum of between-group and within-group variance.
Higher values indicate that observations within the same group tend to resemble one another more strongly under the model.
ICC(2), in commonly used aggregation frameworks, concerns the reliability of group means and depends partly on group size and the underlying ICC.
These statistics address different questions. ICC(1) concerns clustering, while ICC(2) can inform the reliability of aggregated group means.
Neither statistic independently establishes that a construct is theoretically meaningful at the group level.
Is There a Universal ICC or Agreement Threshold for Aggregation?
No single cutoff establishes that aggregation is valid in every research context.
Researchers sometimes use conventional thresholds for ICC or within-group agreement, but their appropriateness depends on the construct, measurement design, group size, analytical purpose, and disciplinary expectations.
A low ICC does not necessarily make a group mean meaningless. For example, a group mean may still be the quantity the research seeks to estimate.
Conversely, high agreement does not establish that respondents adequately represent the group or that the instrument measures the intended construct.
Evidence for aggregation should combine substantive theory, measurement validity, sampling adequacy, and relevant statistical diagnostics.
How Many Individual Respondents Are Needed per Group?
There is no universal minimum number of respondents per group.
The appropriate number depends on the precision required for group summaries, within-group variability, measurement reliability, response rates, and the intended analysis.
For example, a classroom average based on two respondents may be highly sensitive to their particular experiences. An average based on twenty respondents may be more stable, although systematic nonresponse can still produce bias.
Researchers should examine both the number of respondents and the proportion of eligible group members represented.
A large number of respondents does not guarantee representativeness when participation is systematically related to the measured construct.
Unequal group sizes may also produce group means with different precision. Analytical approaches should consider whether these differences matter for the estimand.
How Many Groups Do You Need?
For group-level analysis, the number of distinct groups is a central component of the study's information base.
Suppose 1,000 employees respond to a survey across five organizations. If the research question concerns relationships between organizational characteristics, the analysis contains five organizational cases.
The 1,000 responses can improve measurement of those five organizations, but they do not create additional independent organizations.
With very few groups, estimates of between-group associations may be unstable, sensitive to influential cases, and unable to support complex models.
Researchers should distinguish the number of participants from the number of groups during sample size planning.
The appropriate number of groups depends on the model, expected variation, effect size, and desired precision rather than a universal threshold.
What Is the Difference Between a Group Mean and a Group Effect?
A group mean describes the average value of a variable among members of a group. A group effect concerns a relationship attributable to, or associated with, a group-level characteristic under a specified model.
These should not be conflated.
For example, classrooms with higher mean student motivation may also have higher mean achievement. This is a between-classroom association.
It does not establish that raising classroom motivation would cause achievement to improve.
Nor does it establish the individual-level relationship between motivation and achievement within classrooms.
Group-level causal interpretation requires additional design and identification assumptions beyond calculating group averages.
What Are Ecological and Atomistic Fallacies?
An ecological fallacy occurs when a relationship observed among groups is assumed to describe individuals within those groups.
An atomistic fallacy occurs when an individual-level relationship is assumed to describe relationships between groups.
For example, a positive relationship between classroom average engagement and average achievement does not establish that more engaged students outperform less engaged students within every classroom.
Likewise, a positive individual-level relationship does not guarantee that classrooms with higher average engagement have higher average achievement.
The relationship between group-level data and individual-level inference requires careful interpretation because within-group and between-group relationships can differ.
Should You Aggregate the Data or Use a Multilevel Model?
The choice depends on the research question.
If the principal question concerns group averages or institutional characteristics, aggregation may provide a direct and interpretable analytical dataset.
If the question concerns individual outcomes and group-level influences, a multilevel model may preserve individual variation while accounting for group structure.
For example, a study asking whether classrooms with higher mean engagement have higher mean achievement can analyze classroom summaries.
A study asking whether students' engagement predicts their achievement, and whether that relationship varies between classrooms, requires a different analytical structure.
Multilevel models can separate within-group and between-group components when specified appropriately, but they require sufficient information at the relevant levels.
Researchers should not assume that multilevel modeling is automatically superior to aggregation. Each answers a different estimand.
How Do You Handle Unequal Group Sizes?
Groups may contribute different numbers of individual observations. One classroom may have fifteen respondents, while another has forty.
If each group mean is treated equally, the analysis gives each group equal weight regardless of its size or measurement precision.
If groups are weighted by their number of respondents, larger groups contribute more strongly to the estimated relationship.
Neither approach is universally correct. Equal weighting may be appropriate when the estimand concerns an average group, while size weighting may be appropriate when the estimand concerns an average individual represented through groups.
Precision-based weighting may also be considered when group means have different measurement uncertainty.
Researchers should state which population quantity their weighting strategy targets rather than choosing weights solely for convenience.
What If Some Groups Have Very Few Respondents?
Small respondent counts can make group summaries imprecise and vulnerable to nonresponse or extreme observations.
Researchers may establish minimum coverage requirements, investigate measurement reliability, or use statistical models that account for uncertainty in group estimates.
However, excluding small groups can change the population represented by the analysis.
For example, removing small rural schools may leave a dataset dominated by larger urban schools.
Any exclusion threshold should therefore be justified in relation to measurement quality and the intended target population.
Sensitivity analyses may help determine whether results depend heavily on small or poorly represented groups.
Can Individual Responses Be Used to Study Organizational Culture?
Yes, but researchers should distinguish employees' individual perceptions from an organizational construct.
If organizational culture is conceptualized as a shared set of perceptions or practices, aggregation may require evidence of within-organization agreement and meaningful between-organization variation.
Alternatively, the study may explicitly examine the distribution of employee perceptions within organizations, including disagreement and subgroup differences.
One respondent's answer should not automatically be treated as an objective measure of organizational culture.
Researchers should consider whether multiple informants, organizational documents, direct observations, or other evidence would strengthen measurement validity.
How Does Aggregation Affect Statistical Independence?
Once individual observations are aggregated, each group may contribute one summary to a group-level dataset.
However, independence between groups should not be assumed automatically.
Schools within the same district may share policies. Hospitals within the same network may share management systems. Communities may experience spatially correlated conditions.
Researchers should evaluate clustering and dependence at the group level when relevant.
Aggregation does not eliminate every source of statistical dependence; it changes the level at which dependence must be considered.
Can Aggregation Create Pseudoreplication?
Aggregation itself is not pseudoreplication. Problems arise when researchers misrepresent the number of independent groups or treatment assignments.
For example, if a study compares two instructional interventions assigned to four classrooms, individual student responses do not provide independently randomized treatment replications.
Analyzing the students without accounting for classroom assignment may produce pseudoreplication.
Using classroom summaries may respect the assignment level, although four classrooms may provide very limited information for estimating an intervention effect.
The solution is not merely to aggregate mechanically. The design must provide enough independent units to support the intended comparison.
How Should Group-Level Analysis Be Reported?
Methods sections should explain why the construct is defined at the group level, how individual observations were obtained, and how they were aggregated.
Report the number of groups, respondent counts per group, response coverage, missingness, and relevant evidence supporting measurement reliability or agreement.
Describe whether groups were weighted equally, by size, or by another justified procedure.
Where applicable, distinguish descriptive group summaries from causal claims about group-level influences.
Transparent reporting is consistent with the STROBE emphasis on describing participants, measurement procedures, study size, statistical methods, and limitations in observational research.
