03 · What You Need to Know
Understanding Differential Missingness Across Study Groups
What is differential missingness?
Differential missingness occurs when the frequency, timing, structure, or determinants of missing data differ across study groups.
For example, an intervention group may have more missing final assessments than a comparison group. Alternatively, both groups may have the same overall nonresponse rate but differ in which participants have missing outcomes.
The second situation is easily overlooked. If high-performing students are disproportionately missing from one group and low-performing students from another, equal completion percentages may conceal very different selection processes.
Researchers should therefore distinguish the amount of missing data from its pattern and underlying mechanism.
Why might baseline characteristics produce different missingness patterns?
Study groups may differ in characteristics associated with their ability or willingness to complete measurements.
In observational research, participants may select their exposure according to employment schedules, socioeconomic circumstances, health status, digital access, or motivation. Those characteristics may also affect follow-up participation.
For example, students who voluntarily choose an online learning program may have better internet access than nonparticipants. If follow-up assessments are administered online, the two groups may experience different completion rates even when the assessment procedures are nominally identical.
In randomized studies, baseline characteristics are balanced in expectation under valid random assignment, but chance imbalances can occur. Differences in subsequent missingness may also emerge after assignment.
When planning measurement, researchers should identify variables and procedures most vulnerable to missingness, including whether the risks vary across participant groups.
Can the intervention itself influence whether outcomes are observed?
Yes. Interventions can affect participants' availability, motivation, burden, or willingness to continue providing data.
An intensive learning intervention may require additional activities that compete with follow-up assessments. A difficult or disappointing intervention experience may reduce participants' willingness to return. Conversely, a supportive intervention may increase engagement and make outcome collection easier.
These mechanisms can produce group differences in observation even when the original allocation was randomized.
Importantly, missingness may then be a consequence of assignment or exposure. Conditioning on completion can undermine the comparability created by randomization if completion is also influenced by outcome-related characteristics.
This is one reason an analysis restricted to observed outcomes does not automatically retain all the causal protections of the original randomized design.
How can measurement procedures create unequal missingness?
Different collection procedures can produce different opportunities for nonresponse.
Suppose the intervention group completes assessments during supervised sessions, while the comparison group receives an optional online link. A higher completion rate in the intervention group might reflect the assessment procedure rather than an effect of the instructional approach on participant engagement.
Other potential differences include reminder frequency, assessment accessibility, language, device requirements, staff availability, and the length of the measurement window.
Researchers should aim for comparable outcome ascertainment procedures where scientifically and operationally appropriate.
When identical procedures are impossible, document the differences and consider how they may affect missingness and measurement comparability.
Why does outcome-related missingness create particular concern?
Missingness becomes especially consequential when the probability of observing an outcome depends on that outcome or on factors strongly related to it.
For example, students expecting poor examination performance may be less likely to complete an optional assessment. If that tendency differs across instructional groups, the observed outcomes may not represent the outcomes of all enrolled participants.
Similarly, participants experiencing adverse effects may discontinue a clinical intervention and become less likely to provide follow-up measurements.
The resulting observed sample may differ systematically from the population relevant to the estimand.
Whether the bias increases or decreases the apparent intervention effect depends on the direction and strength of the selection mechanisms. Differential missingness should not automatically be described as favoring one group.
How do MCAR, MAR, and MNAR apply to group differences?
Missing completely at random (MCAR), missing at random (MAR), and missing not at random (MNAR) describe relationships between missingness and the observed or unobserved data.
Group assignment can be included among the observed variables used to characterize those relationships.
| Situation |
Possible interpretation |
Important qualification |
| Missingness differs by randomized group but not by unobserved outcomes after conditioning on recorded information. |
May be compatible with MAR conditional on group and other observed predictors. |
MAR is an assumption, not something established by observing group differences. |
| Missingness depends on baseline achievement, which is fully observed. |
May be compatible with MAR when achievement and other relevant predictors are included. |
The adequacy of the observed information must still be justified. |
| Missingness depends on an unobserved outcome even after conditioning on available variables. |
Represents an MNAR mechanism relative to the observed information. |
The dependence generally cannot be identified from observed data alone. |
| Missingness arises from an independent random technical failure. |
May be compatible with MCAR. |
The independence assumption must be credible for the relevant data-generating process. |
Different observation rates across randomized groups are evidence against an unconditional MCAR mechanism for the combined dataset when group assignment is part of the relevant full-data structure. They do not, by themselves, establish MNAR.
Likewise, equal missingness rates do not establish MCAR. The determinants of observation may differ even when the percentages coincide.
How can conditioning on complete cases introduce selection bias?
Complete-case analysis includes only observations with all variables required by the specified analysis.
Suppose intervention assignment affects assessment completion, while baseline motivation also affects completion and examination performance. Completion is then a common effect of assignment and motivation.
Restricting analysis to completers may create an association between assignment and motivation, even if the two were independent before selection.
This is a form of collider-related selection bias. The causal problem arises through conditioning on completion, not necessarily through including completion as an explicit regression covariate.
Understanding how conditioning on colliders can introduce bias is therefore relevant to missing-outcome analyses.
Complete-case analysis is not universally biased, but its validity depends on assumptions concerning the observation process and the estimator.
What should researchers compare across groups?
Begin with a descriptive account of the available data. Report the number eligible for each measurement, the number observed, and the number missing within each group.
For longitudinal studies, examine completion at each scheduled occasion and whether participants return after missing an earlier assessment.
Compare observed baseline characteristics between participants with and without follow-up outcomes, both overall and within study groups where informative.
Useful variables may include baseline outcome values, prior achievement, age, study site, access barriers, and other predictors relevant to the observation process.
These comparisons can reveal systematic differences involving observed characteristics. They cannot determine whether missingness also depends on the unobserved outcome.
Researchers should avoid relying exclusively on significance tests for differences between completers and noncompleters. Descriptive magnitudes and substantive relevance are often more informative.
Should missingness be summarized only as one overall percentage?
No. An overall percentage can conceal important differences across groups, variables, and measurement occasions.
For example, a study may report 15% missing outcomes overall, but that figure could arise from 5% missingness in one group and 25% in another when group sizes are equal.
Even group-specific percentages may be insufficient when missingness occurs at different times or among participants with different baseline characteristics.
Researchers should consider tables showing observed and missing counts by group and occasion, alongside relevant baseline comparisons and documented reasons for nonresponse.
For repeated measures, visualizing individual response patterns may help distinguish intermittent missingness from monotone dropout, in which participants stop providing subsequent observations.
Can differential missingness change the target population of an analysis?
Yes. An analysis limited to observed outcomes may effectively describe participants who remained observable rather than the original eligible or randomized population.
This distinction is particularly important when the probability of observation depends on exposure, prognosis, or the outcome itself.
For example, a randomized study may aim to estimate the effect of assignment among all randomized participants. An unadjusted complete-case comparison instead uses only participants with observed outcomes, and that selected comparison may not identify the intended effect.
The issue is not simply a smaller sample. It is whether the selected observations still support the original causal contrast.
Researchers should distinguish the estimand from the analytical sample and explain what assumptions connect the two.
Which analytical methods can address group-specific missingness?
Several approaches may be appropriate, depending on the estimand, available data, and missingness assumptions.
Multiple imputation can incorporate group assignment and observed predictors of missingness and outcomes. Inverse probability weighting can account for modeled probabilities of observation. Likelihood-based approaches may use incomplete longitudinal records under suitable assumptions.
These methods should accommodate important differences between groups in the outcome and observation processes. For example, an imputation model that imposes identical outcome relationships across groups may be inappropriate when the intervention changes those relationships.
However, no method automatically resolves MNAR mechanisms. Sensitivity analyses may be needed to examine plausible differences between observed and unobserved outcomes.
Researchers should also recognize that methods can become unstable when observation probabilities are very low in particular groups or covariate combinations.
How can research design reduce differential missingness?
Prevention should begin before recruitment and continue throughout follow-up.
Where feasible, use comparable outcome collection procedures across groups, accessible assessment modes, appropriate reminder schedules, and consistent definitions of completion.
Anticipate whether intervention demands or participant characteristics could make one group more difficult to follow.
Preserve outcome collection after intervention discontinuation when consistent with the estimand, consent, and ethical approval.
Document reasons for missingness without assuming that stated reasons establish the statistical mechanism.
These measures form part of planning for missing data during research design, rather than relying exclusively on statistical procedures after collection ends.
