How Do You Compare Quantitative and Qualitative Findings Collected at Different Levels of Analysis?

Can Findings About Individuals Be Compared With Findings About Groups or Institutions?

You are investigating the implementation of an online learning program. Your quantitative analysis examines student engagement and academic performance across several university departments. Your qualitative interviews, however, involve department heads describing institutional policies, instructional expectations, and implementation challenges.

Both components concern the same educational program. But do they provide evidence at the same analytical level?

The quantitative results describe students and perhaps departmental averages. The interviews describe organizational decisions and institutional conditions. If departments with higher engagement also report stronger administrative support, can you conclude that institutional support explains students' engagement?

Not necessarily. The findings may be meaningfully related, but the relationship requires careful consideration of analytical levels, sampling, aggregation, and alternative explanations.

The challenge is to integrate evidence about different units without treating group-level patterns as individual-level relationships or assuming that individual experiences adequately represent entire institutions.

Compare Findings Across Levels Only When Their Relationship Is Explicitly Justified

Understanding Cross-Level Integration in Mixed-Methods Research

What Is a Level of Analysis?

The level of analysis refers to the unit about which a researcher develops an analytical claim. Depending on the study, this may be an individual, classroom, department, institution, community, or wider system.

For example, a student-level analysis may investigate relationships between individual engagement scores and academic performance. A classroom-level analysis may examine differences between average engagement scores across classes. An institutional analysis may investigate policies or organizational practices across universities.

These levels are related but not interchangeable.

A classroom average summarizes observations about students within a classroom. It does not establish that every student has the average characteristic or that relationships between classroom averages also exist among individuals.

Researchers should therefore specify what each finding describes before attempting integration.

How Is the Level of Analysis Different From the Level of Data Collection?

Data collection and analysis may occur at different levels.

For example, researchers may interview individual teachers about departmental policies. The interviews are collected from individuals, but the resulting analysis may concern organizational processes.

Likewise, researchers may collect student-level assessment scores and aggregate them into departmental averages for comparison.

The original observations remain individual-level data, while the derived summaries represent departments.

The distinction matters because the level at which data were collected does not automatically determine the level at which conclusions are justified.

Researchers must examine whether the evidence adequately supports the intended unit of inference. One teacher's account may provide important evidence about an institutional process without establishing that the account represents every teacher or the institution as a whole.

What Types of Cross-Level Comparisons Are Common?

Mixed-methods studies frequently combine evidence from different analytical levels, particularly in education, organizational research, and implementation studies.

Each arrangement can support meaningful integration, but the strength and scope of the interpretation depend on the evidence linking the levels.

What Is the Ecological Fallacy?

The ecological fallacy occurs when relationships observed among aggregate units are incorrectly assumed to hold among individuals within those units.

Robinson (1950) demonstrated why ecological correlations cannot generally be interpreted as individual-level correlations.

Consider a hypothetical study finding that departments with higher average platform participation also have higher average academic performance.

This does not establish that students who participate more frequently necessarily obtain higher scores within each department.

The aggregate relationship may reflect differences in student composition, assessment practices, institutional resources, or other characteristics.

Qualitative interviews with department heads may illuminate some of these conditions, but they do not automatically establish the individual-level relationship.

Researchers should distinguish the aggregate pattern from claims about individual students.

What Is the Individualistic Fallacy?

The individualistic fallacy, sometimes discussed in relation to atomistic inference, occurs when relationships or characteristics observed among individuals are incorrectly generalized to groups or higher-level systems.

For example, interviews with several teachers may reveal that they experience uncertainty about institutional generative AI policies.

Those accounts do not establish that the institution itself has no clear policy. The policy may exist but be poorly communicated, inconsistently interpreted, or unevenly implemented.

Similarly, a relationship between individual teachers' confidence and implementation frequency does not necessarily establish that departments with higher average confidence implement AI more frequently.

Cross-level integration requires evidence about the higher-level construct rather than assuming that individual accounts or relationships automatically scale upward.

Why Can Aggregate and Individual Findings Differ?

Aggregate findings summarize information across units, potentially concealing substantial within-group variation.

For example, two departments may have similar average engagement scores even though one contains students with consistently moderate engagement and the other contains students with widely contrasting engagement profiles.

Qualitative interviews may reveal experiences that help interpret these differences, but the interview sample must be relevant to the groups being compared.

Aggregate and individual-level relationships can also differ because of compositional effects, contextual influences, and statistical aggregation.

Researchers should therefore avoid assuming that a disagreement between individual accounts and aggregate statistics necessarily indicates a methodological error.

The findings may describe genuinely different properties of the phenomenon.

How Do You Establish the Relationship Between Analytical Levels?

Begin by identifying the units represented in each component and the organizational or contextual relationships connecting them.

For example, students may be nested within classes, classes within departments, and departments within universities.

If qualitative interviews concern teachers, researchers should determine which classes or departments those teachers represent and whether those contexts correspond with the quantitative observations.

A clear mapping may make it possible to relate student-level outcomes to classroom-level implementation conditions.

However, a shared institution alone may provide insufficient linkage for particular explanatory claims.

The researcher should document which units correspond, which do not, and what contextual relationships remain uncertain.

Should Quantitative Data Be Aggregated to Match Qualitative Findings?

Aggregation may be appropriate when the overarching question concerns group-level or institutional differences.

For example, individual student outcomes might be summarized by department to compare departmental patterns with qualitative accounts from department heads.

However, aggregation changes the analytical question and may discard important within-group variation.

Researchers should justify the aggregation procedure, examine group sizes and variability, and consider whether the aggregate measure represents the intended construct.

For perceptual constructs, agreement among individuals may also be relevant before interpreting a group average as a shared organizational characteristic.

Aggregation should not be performed solely to make two datasets appear comparable.

Can Multilevel Statistical Analysis Help?

When quantitative data have a hierarchical structure, multilevel statistical models may help distinguish individual-level and group-level variation.

For example, students may be nested within courses, and courses may differ in instructional practices.

A multilevel model can represent relevant levels explicitly and, where supported by the data and assumptions, estimate associations involving individual and contextual characteristics.

Raudenbush and Bryk (2002) provide a foundational treatment of hierarchical linear models and their application to nested data.

However, multilevel modeling does not automatically establish causal relationships or solve all problems of cross-level integration.

The number of higher-level units, measurement quality, model specification, and sampling design remain important.

Qualitative findings may help interpret contextual processes, but they should not be treated as statistical validation of the model.

How Do You Compare Findings Without Statistical Aggregation?

Statistical aggregation is not always necessary.

Researchers may integrate findings through case-oriented comparison, contextual analysis, or joint displays organized around relevant settings.

For example, quantitative student outcomes could be summarized separately for selected courses, while interviews with teachers and students describe the learning conditions in those courses.

The integrated analysis might examine whether particular contextual accounts are compatible with the observed patterns.

Such comparisons can be analytically informative without requiring a formal multilevel statistical model.

However, the researcher should avoid presenting contextual correspondence as evidence of causation when alternative explanations have not been addressed.

How Can Joint Displays Support Cross-Level Integration?

A joint display can organize findings according to the units or contexts that connect the components.

Guetterman, Fetters, and Creswell (2015) describe joint displays as a means of supporting integrated mixed-methods analysis.

For cross-level research, rows may represent classrooms or departments, with columns presenting aggregate quantitative findings, qualitative accounts, and integrated interpretations.

A display should also identify the analytical level of each finding so that readers do not confuse individual experiences with institutional characteristics.

These examples are hypothetical. The display illustrates possible analytical relationships, not established explanations.

What If Qualitative Findings Contradict Aggregate Quantitative Results?

Apparent contradiction may arise because the components describe different levels or aspects of the phenomenon.

For example, an institution may report high overall satisfaction while interviewed students describe substantial difficulties.

Both findings may be credible if the interview participants experienced difficulties that are uncommon, concentrated in particular subgroups, or insufficiently represented by the overall satisfaction measure.

The researcher should examine the sample composition, distribution of quantitative responses, and relevance of the qualitative cases.

Where findings remain genuinely inconsistent, the disagreement should be reported and interpreted transparently.

The procedures for investigating conflicting mixed-methods findings are particularly important when analytical levels differ.

Can Cross-Level Integration Establish a Causal Explanation?

Not automatically.

Suppose departments with stronger reported instructional support also have higher average student performance.

Interviews may suggest plausible ways in which support influences teaching practices, but the combined evidence does not by itself establish that institutional support caused the observed differences.

Departments may differ in student characteristics, resources, assessment practices, or other relevant conditions.

Cross-level causal inference requires appropriate design features, temporal ordering, relevant assumptions, and evidence addressing alternative explanations.

Researchers should distinguish contextual interpretation from causal identification.

How Should Cross-Level Integration Be Reported?

The methodology should specify the units of data collection and analysis for each component.

Researchers should explain how cases, groups, or institutions were linked and whether quantitative information was aggregated.

When multilevel statistical procedures were used, the report should describe the relevant model structure and assumptions.

The integrated findings should identify whether conclusions concern individuals, groups, institutions, or relationships across levels.

Limitations arising from unmatched cases, small numbers of groups, or insufficient contextual linkage should be acknowledged.

The final interpretation should not silently shift between levels of inference.

Integrating Student Engagement Data With Departmental Implementation Accounts

Suppose departments with higher average platform participation also have leaders describing more structured online learning requirements.

The findings may suggest a contextual relationship worth further investigation.

However, the researcher cannot conclude that individual students participated more because of those requirements without additional evidence addressing alternative explanations.

With only four departments, statistical inference about departmental-level relationships would also be particularly limited.

The defensible contribution is a contextually informed interpretation of departmental patterns, with clear boundaries around individual-level and causal inference.

Common Mistakes When Integrating Findings Across Analytical Levels

How to Integrate Different Analytical Levels Without Overclaiming

Begin by identifying what each dataset describes and which units the findings represent.

Then establish the relationship between those units and decide whether integration should occur through case mapping, group-level comparison, contextual interpretation, or appropriate multilevel analysis.

A joint display organized around explicitly defined units can make the reasoning transparent.

The objective is not to force all evidence into one analytical level. It is to develop a coherent interpretation that preserves the differences between the units represented.

Check Whether Your Cross-Level Integration Is Defensible

Frequently Asked Questions About Cross-Level Mixed-Methods Integration

Preserve Analytical Levels When Integrating Different Forms of Evidence

How to Cite This Guide

This guide is intended to be read, shared, and used in research, teaching, and academic work. If you draw on its ideas, explanations, or other content, please acknowledge the source by citing the guide. Doing so gives appropriate credit and helps your readers locate the original resource.