03 · What You Need to Know
Understanding and Investigating Mixed-Methods Disagreement
What Does Disagreement Mean in Mixed-Methods Research?
Disagreement occurs when quantitative and qualitative findings appear to support different or incompatible interpretations of a research problem.
For example, a questionnaire may indicate that students generally perceive academic feedback positively, while interview participants describe feedback as insufficient or difficult to use.
These findings may represent a genuine contradiction, but that conclusion should not be reached immediately.
The questionnaire may measure overall satisfaction, while the interviews explore particular experiences. The samples may differ, or the evidence may have been collected at different times.
Disagreement should therefore be treated as an analytic question rather than an automatic indication that one strand is incorrect.
What Is the Difference Between Convergence, Complementarity, and Divergence?
| Relationship |
Meaning |
Illustrative example |
| Convergence |
Findings provide broadly compatible evidence concerning a related phenomenon. |
Survey respondents report valuing timely feedback, while interview participants describe prompt responses as helpful. |
| Complementarity |
Findings illuminate different but related dimensions of a phenomenon. |
A survey estimates reported access to support, while interviews describe how students obtain assistance. |
| Apparent divergence |
Findings initially seem inconsistent, but may concern different constructs, contexts, or levels. |
High satisfaction with online learning coexists with complaints about one support service. |
| Substantive divergence |
Findings concerning sufficiently comparable phenomena support materially different interpretations. |
A survey indicates widespread perceived accessibility of a service, while a comparable qualitative sample describes persistent barriers to accessing that same service. |
These categories are interpretive judgments, not automatic classifications. Researchers should explain why a particular relationship is assigned.
The distinction is especially useful when integrating quantitative and qualitative evidence during analysis.
Why Do Quantitative and Qualitative Findings Sometimes Disagree?
Disagreement can arise because the strands measure or interpret different dimensions of the phenomenon, represent different participants, or capture experiences under different circumstances.
It may also reflect limitations in questionnaire items, interview procedures, sampling, statistical precision, or qualitative interpretation.
Sometimes the disagreement is substantive. Participants may simultaneously hold positive general attitudes toward a service while experiencing serious difficulties with particular aspects of it.
For example, students may appreciate the flexibility of online learning while struggling with communication, workload, or access to academic assistance.
Such experiences are not necessarily contradictory. They may reveal that the phenomenon is more differentiated than the initial research model assumed.
First, Check Whether the Strands Examine the Same Construct
Conceptual comparability is one of the first issues to investigate.
Suppose the quantitative strand measures perceived availability of academic support, while the qualitative strand examines whether students receive useful assistance when needed.
Availability and experienced accessibility are related but distinct constructs.
A university may provide several formal support channels while students still experience difficulties obtaining timely responses.
Consequently, favorable quantitative ratings of availability may coexist with qualitative accounts of inaccessible support without creating a direct contradiction.
Researchers should inspect the actual questionnaire items, interview prompts, coding definitions, and qualitative interpretations rather than relying on similar construct labels.
Second, Examine Sampling Differences
Quantitative and qualitative strands may represent different participants or populations.
A survey may include hundreds of students across several programs, while interviews focus on a purposively selected group with particular experiences.
If interview participants were selected because they encountered difficulties, their accounts should not be expected to reproduce the overall survey distribution.
Likewise, participants who volunteer for interviews may differ systematically from those who do not.
Researchers should examine eligibility criteria, recruitment procedures, participant characteristics, nonresponse, and the relationship between samples.
These considerations are particularly important when deciding how quantitative and qualitative samples should be connected.
Third, Check the Unit of Analysis
Mixed-methods disagreement may arise when the strands operate at different levels.
For example, quantitative results may describe average institutional performance, while qualitative interviews concern individual teachers' experiences.
An institution with favorable average outcomes can still contain individuals experiencing substantial difficulties.
Researchers should avoid assuming that aggregate relationships necessarily apply to individuals. This is related to the ecological fallacy.
The reverse problem can also occur when individual accounts are used to make unsupported claims about entire institutions or populations.
Before declaring disagreement, establish what each finding actually describes.
Fourth, Consider Timing and Context
Quantitative and qualitative evidence collected at different times may reflect changing conditions.
For example, a survey conducted before the introduction of a new learning management system may produce different findings from interviews conducted after implementation.
Institutional policies, teaching arrangements, assessment schedules, or technical infrastructure may have changed between collection periods.
Researchers should document relevant contextual developments and assess whether temporal differences affect comparability.
In sequential designs, different collection periods are often intentional. Nevertheless, their implications should be considered when integrating findings.
Fifth, Revisit Quantitative Measurement and Analysis
When quantitative results appear inconsistent with qualitative evidence, researchers should examine the measurement and analysis procedures.
Relevant questions include whether the instrument adequately represents the intended construct, whether items were interpreted as expected, and whether the statistical model was appropriate.
Researchers should also consider missing data, measurement reliability, sample characteristics, effect estimates, and statistical uncertainty.
For example, a statistically nonsignificant association does not necessarily establish that no relationship exists. An imprecise estimate may be compatible with several substantively different effects.
Similarly, a statistically significant result may have limited practical importance.
Any additional statistical analyses prompted by disagreement should be identified as exploratory when they were not planned in advance.
Researchers should not repeatedly modify models until the results align with the qualitative findings.
Sixth, Revisit Qualitative Interpretation
Qualitative findings should also be critically examined.
Researchers may need to revisit transcripts, field notes, coding decisions, and the development of themes or categories.
For example, an interpretation that students generally distrust institutional support may be based on particularly vivid negative accounts rather than the broader pattern of meanings across the dataset.
Researchers should examine whether the interpretation adequately represents variation, context, and potentially contradictory accounts.
However, qualitative findings should not be reduced to frequency counts merely to make them resemble survey results.
In interpretive approaches, a theme's significance does not necessarily depend on how many participants explicitly mention it.
The appropriate review procedures should remain consistent with the selected qualitative methodology.
Can Social Desirability or Response Processes Explain Disagreement?
Potentially. Participants may respond differently depending on the research setting, perceived expectations, anonymity, interviewer relationships, or question wording.
For example, students may provide favorable questionnaire ratings while discussing difficulties more openly during confidential interviews.
The reverse is also possible. Participants may express stronger criticism in a survey than in an interview with someone perceived as institutionally connected.
However, researchers should not assume social desirability simply because the findings disagree.
Such an explanation requires supporting evidence concerning response processes, study conditions, or participant accounts.
Does Disagreement Mean One Dataset Is Wrong?
No. Both strands may accurately represent different dimensions or contexts.
Alternatively, one or both may contain methodological limitations that affect interpretation.
The appropriate response is to evaluate the evidence rather than assign automatic priority based on whether it is quantitative or qualitative.
Quantitative findings are not inherently more objective simply because they contain numerical estimates. Qualitative accounts are not automatically more authentic because they provide detailed descriptions.
Each strand should be evaluated according to the claims it is intended to support.
Should You Collect Additional Data to Investigate Disagreement?
Additional data collection may be useful when the discrepancy is important to the research question and the study can accommodate further inquiry.
For example, researchers might conduct follow-up interviews to clarify how participants interpreted a questionnaire item or examine institutional documents relevant to an unexpected finding.
However, further collection is not always necessary or feasible.
Researchers should consider whether the additional evidence would meaningfully distinguish among competing explanations, whether participants can be contacted appropriately, and whether ethics approval permits the proposed procedures.
Collecting more data merely to obtain agreement is not a defensible objective.
How Can a Joint Display Help Investigate Disagreement?
A joint display can organize quantitative findings, qualitative interpretations, possible explanations, and unresolved questions in a common structure.
For example, a row may contain a favorable survey result alongside qualitative accounts of difficulties with the same service.
The researcher can then identify whether the apparent disagreement concerns construct differences, sampling variation, contextual conditions, or a genuine tension in the evidence.
A useful display may also include a column recording the strength or limitations of each proposed explanation.
Planning a joint display for mixed-methods integration can help ensure that disagreement is examined systematically rather than discussed selectively.
Should You Reanalyze the Data Until the Findings Converge?
No. Reanalysis may be justified when it addresses a methodological concern, tests a plausible alternative interpretation, or corrects an identified error.
However, repeatedly modifying coding decisions, excluding participants, or changing statistical models solely to produce agreement introduces serious risks of selective analysis.
Researchers should preserve the distinction between planned analyses, justified corrections, and exploratory investigations prompted by unexpected results.
Consequential changes should be documented, including why they were made and how they affected the findings.
Agreement should be an empirical possibility, not a predetermined requirement.
Can Divergence Be a Valuable Research Finding?
Yes. Divergence may reveal that the original research question or conceptual framework overlooks important dimensions of the phenomenon.
For example, a survey may indicate that students have access to digital learning resources, while interviews reveal difficulties using those resources effectively.
The combined evidence may suggest that resource availability and practical usability should be distinguished.
This does not mean every contradiction is theoretically significant. Some arise from weak measurement, sampling limitations, or incompatible data.
The contribution depends on whether the researcher can establish a credible explanation or identify a defensible unresolved question.
How Should Unresolved Disagreement Be Reported?
Researchers should describe the findings from both strands accurately and explain why the discrepancy remains unresolved.
For example, a report might state that the survey indicated generally favorable perceptions of institutional support, while interviews revealed substantial difficulties among selected participants.
The researcher could then explain that differences in sampling and construct specificity limit direct comparison.
Where the evidence cannot establish why the findings differ, that uncertainty should be acknowledged.
An unresolved discrepancy is preferable to an unsupported claim that the strands confirm one another.
How Does Disagreement Affect Meta-Inferences?
Meta-inferences are conclusions developed through integrating quantitative and qualitative evidence.
When findings diverge, the meta-inference may concern the limits of an initial interpretation rather than a single reconciled explanation.
For example, the combined evidence may suggest that high average satisfaction ratings do not adequately describe the experiences of all student groups.
However, researchers should avoid inferring the prevalence of particular qualitative experiences without suitable quantitative evidence.
The meta-inference should remain appropriately bounded by sampling, measurement, and contextual limitations.
What If Disagreement Undermines the Original Research Hypothesis?
The researcher should report the evidence rather than protect the original hypothesis.
Mixed-methods inquiry can reveal that an expected relationship is weaker, more conditional, or conceptually different from what was anticipated.
For example, qualitative accounts may suggest that the meaning of institutional support varies substantially among students, complicating the interpretation of a single quantitative support score.
The appropriate conclusion may be that the original model requires reconsideration, not that one dataset should be excluded.
Any revised theoretical explanation should be presented as supported, tentative, or exploratory according to the available evidence.
How Should Disagreement Be Discussed in a Journal Article?
The report should identify the specific findings that differ, explain the basis for comparing them, and examine plausible explanations.
Researchers should distinguish empirical observations from interpretations developed during integration.
They should also explain whether the disagreement changes the study's substantive conclusions or reveals limitations in measurement, sampling, or design.
Where appropriate, a joint display can make the discrepancy and its interpretation more transparent.
Simply stating that the findings were inconsistent, without examining why, provides limited methodological insight.
Watch Out
Do not selectively revise, exclude, or reinterpret findings merely to make quantitative and qualitative results agree. Convergence is not a requirement of mixed-methods rigor, and unexplained disagreement should be reported rather than concealed.
