03 · What You Need to Know
How Missing Information Can Distort Research Findings
What Counts as Missing Data in Research?
Missing data occur when a value required for an intended analysis is unavailable for an observation, participant, variable, or measurement occasion.
Missingness may affect outcomes, exposures, baseline covariates, repeated measurements, or other variables used in the study.
For example, a student may complete an examination but leave several questionnaire items unanswered. Another student may provide baseline information but miss every follow-up assessment.
In an institutional dataset, some demographic or academic variables may be absent because they were not collected consistently across departments.
These situations differ methodologically. Missing outcome data may directly compromise estimation of an intervention effect, while missing baseline covariates may complicate adjustment for confounding.
Researchers should identify precisely which information is missing before deciding how to analyze the dataset.
Why Does Missingness Matter for Internal Validity?
Internal validity concerns whether a study supports the causal interpretation being drawn from its comparison.
Suppose researchers want to estimate the effect of assigning students to an educational intervention. The relevant comparison may concern all students originally assigned to each condition.
If outcome information is available only for selected participants, the observed group means may not represent the outcomes of the original groups.
For example, students with poor outcomes may be less likely to provide follow-up data. The mean among observed participants could then overstate the mean that would have been obtained if everyone had been assessed.
Missingness may also affect confounding adjustment when important baseline characteristics are unavailable.
However, not every missing value creates systematic bias. Under some missingness mechanisms and analytical conditions, estimates may remain unbiased while becoming less precise.
The key is to evaluate whether the observed information and analytical assumptions support the intended causal effect.
What Is the Difference Between MCAR, MAR, and MNAR?
Missing-data methodology commonly distinguishes three mechanisms: missing completely at random, missing at random, and missing not at random.
These terms describe statistical relationships between missingness and data values. They do not simply describe whether participants had a reasonable explanation for failing to respond.
| Mechanism |
Definition |
Illustrative situation |
| Missing completely at random (MCAR) |
Missingness is independent of the observed and unobserved data values under the relevant model. |
A random equipment failure prevents measurement for an otherwise randomly selected subset of participants. |
| Missing at random (MAR) |
Conditional on appropriately observed variables, missingness does not additionally depend on the missing values. |
Follow-up response depends on recorded baseline achievement, and missingness is independent of the unobserved follow-up score after conditioning on the relevant observed information. |
| Missing not at random (MNAR) |
Missingness continues to depend on unobserved values after conditioning on the observed information included in the model. |
Students with particularly poor unobserved final scores are less likely to submit results, even after accounting for measured predictors. |
These examples are hypothetical and depend on the stated assumptions. An equipment failure, for instance, is not MCAR if it occurs disproportionately in settings associated with the outcome.
Similarly, missingness associated with baseline achievement is not necessarily MNAR if baseline achievement is observed and adequately accounts for the relevant relationship.
MAR is a conditional assumption. It does not mean that missingness occurs randomly in everyday language.
Researchers should not claim to have established MAR or MNAR merely by comparing participants with and without observed outcomes.
Can Researchers Determine the Missingness Mechanism From the Observed Data?
Observed data can provide useful evidence about missingness, but they generally cannot establish whether missing values depend on their own unobserved values.
Researchers may examine whether missingness is associated with measured characteristics such as baseline achievement, age, intervention assignment, or previous outcomes.
Such associations can challenge the plausibility of MCAR.
However, observing no significant association does not prove MCAR. Statistical tests may have limited power, and unmeasured relationships may remain.
Likewise, MAR and MNAR cannot generally be distinguished conclusively using observed data alone because their difference concerns dependence on information that is missing.
Researchers must therefore combine observed evidence with substantive knowledge about the study setting and the reasons information was not collected.
Why Does the Type of Missing Variable Matter?
Missing outcome data and missing covariate data can affect causal analysis through different mechanisms.
If outcomes are missing, researchers may not observe the values needed to estimate the effect directly.
If confounders are missing, adjustment may be incomplete or the analysis may exclude participants whose baseline information is unavailable.
Missing exposure information may introduce uncertainty about treatment classification or the exposure contrast.
Longitudinal studies may contain intermittent missing measurements, monotone dropout patterns, or both.
For example, one student may miss the second assessment but return for the third. Another may leave after the first assessment and provide no further observations.
These patterns can influence which methods are appropriate and what assumptions are needed.
The distinction between missing outcome values and participant loss is especially important when examining differential attrition between study groups.
Why Can Complete-Case Analysis Be Misleading?
Complete-case analysis includes only observations with all variables required for the analysis.
It is straightforward to implement and may be appropriate under certain missingness and model conditions.
However, it can reduce sample size and produce biased estimates when inclusion in the complete-case dataset is related to the outcome or causal relationships being estimated.
Suppose students with lower academic achievement are less likely to provide final examination scores.
If the analysis excludes these students, the observed mean may overrepresent stronger performers.
In a randomized trial, complete-case analysis may also undermine the intended assignment comparison when outcome observation depends on treatment and potential outcomes.
The validity of complete-case analysis is not determined solely by the percentage of missing values. It depends on the missingness mechanism, estimand, and analytical model.
Does Missingness Always Bias Estimates Toward Better Outcomes?
No. The direction of bias depends on which observations are missing and how missingness relates to outcomes and treatment conditions.
If participants with poor outcomes are disproportionately missing, observed results may appear more favorable.
If participants with strong outcomes are more likely to leave, the observed results may appear less favorable.
Different missingness processes across comparison groups can also exaggerate, diminish, or reverse an estimated treatment effect.
Researchers should avoid assuming that missing data necessarily favor the intervention.
What Is Multiple Imputation?
Multiple imputation is a statistical approach that creates several completed datasets by replacing missing values with plausible draws from an imputation model.
Each completed dataset is analyzed using the planned substantive model. The resulting estimates are then combined using appropriate rules that account for within-imputation and between-imputation uncertainty.
Unlike single imputation, multiple imputation is intended to reflect uncertainty about the missing values rather than treating one replacement as known truth.
It can support valid inference when the imputation model is suitable and the required missingness assumptions are defensible.
However, multiple imputation does not reveal the actual missing values and does not automatically remove bias from unmeasured outcome-dependent missingness.
Including relevant predictors of missingness and outcomes may make an MAR assumption more plausible, but cannot establish it conclusively.
Why Is Replacing Missing Values With the Mean Often Problematic?
Mean imputation replaces missing observations with an observed average.
Although simple, this procedure can distort the distribution of a variable, reduce apparent variability, and underestimate uncertainty.
It may also distort relationships between variables and produce misleading standard errors or causal estimates.
For example, replacing every missing examination score with the observed group mean assumes away potentially important differences between students with observed and missing outcomes.
The resulting dataset may appear complete while concealing uncertainty about the unobserved values.
Researchers should not treat mean imputation as a generally adequate solution merely because it permits conventional statistical software to run an analysis.
What About Last Observation Carried Forward?
Last observation carried forward replaces a missing follow-up value with the participant's most recent observed value.
This method may be attractive in longitudinal research because it appears to preserve earlier information.
However, it imposes a strong assumption that the unobserved outcome remains equal to the previous observation.
That assumption may be implausible when participants are expected to improve, deteriorate, recover, or otherwise change over time.
Last observation carried forward can distort estimated effects and uncertainty, and it should not be treated as a neutral or automatically conservative method.
Its suitability requires specific justification rather than routine application.
Can Likelihood-Based Methods Handle Missing Longitudinal Outcomes?
Likelihood-based approaches, including appropriately specified mixed-effects models, can use available repeated measurements without requiring every participant to have a complete observation sequence.
Under suitable assumptions, including an appropriate MAR mechanism and correct model specification, these approaches may provide valid inference.
However, a mixed-effects model does not automatically eliminate missing-data bias.
If outcome observation depends on unobserved values in ways not addressed by the model, estimates may remain biased.
Researchers should therefore distinguish the ability of a method to analyze incomplete datasets from the validity of its assumptions about missingness.
Can Inverse Probability Weighting Address Missing Outcomes?
Inverse probability weighting assigns greater weight to observed participants who represent others with similar measured characteristics but lower probabilities of being observed.
For example, if students with particular baseline characteristics are less likely to complete follow-up, observed students with those characteristics may receive larger weights.
The method can support valid estimation when relevant predictors of observation and outcomes are adequately measured, observation probabilities are appropriately modeled, and positivity assumptions hold.
Extremely small estimated observation probabilities can produce unstable weights and imprecise estimates.
Like multiple imputation, weighting does not automatically resolve missingness that depends on unmeasured outcome-related factors.
Why Are Sensitivity Analyses Important?
Because missingness assumptions cannot generally be verified from observed data alone, researchers should examine whether their conclusions change under plausible alternative assumptions.
A sensitivity analysis may consider scenarios in which missing participants have systematically better or worse outcomes than predicted under the primary model.
For example, researchers might examine how an estimated intervention effect changes if missing intervention participants are assumed to score several points below their model-predicted values.
Pattern-mixture models, selection models, and delta-adjustment approaches can support structured sensitivity analyses, depending on the data and estimand.
The assumptions and numerical departures should be justified using substantive knowledge rather than chosen solely to produce a preferred result.
Sensitivity analysis does not prove which scenario is true. It reveals how dependent the conclusion is on assumptions about unobserved information.
How Can Missing Data Interact With Other Internal Validity Threats?
Missingness rarely occurs in complete isolation from the research design.
Participants may be missing because they discontinue an intervention, experience adverse outcomes, face access barriers, or become unavailable during follow-up.
For example, students who receive an intervention but find it difficult may both fail to adhere to the assigned activities and omit their final assessments.
These events are related but analytically distinct. Nonadherence changes treatment receipt, while missing outcome data reduce information about the resulting outcomes.
Likewise, preexisting differences between comparison groups may influence both outcome values and the probability that those values are observed.
Researchers should consider these relationships when developing a causal model and selecting analytical methods.
How Can Researchers Prevent Missing Data?
Preventing avoidable missingness is generally preferable to relying entirely on analytical correction.
The National Research Council's report on missing data in clinical trials emphasizes careful study design, continued outcome collection, and analyses grounded in explicit assumptions.
Practical measures include reducing unnecessary assessment burden, scheduling accessible follow-up procedures, maintaining appropriate contact information, and planning alternative methods of outcome collection.
In educational research, institutional assessment records may sometimes provide outcomes for students who discontinue active participation, subject to consent, ethical approval, and applicable privacy requirements.
Researchers should identify critical variables before data collection and establish procedures for detecting missing or inconsistent entries promptly.
Retention procedures must respect participant autonomy and should not create coercive pressure to remain in a study.
How Should Missing Data Be Reported?
Researchers should report the amount and pattern of missingness for variables central to the analysis.
For intervention studies, outcome availability should be described separately by assigned group and relevant follow-up point.
Reports should identify known reasons for missingness, relevant characteristics associated with observation, and the methods used to handle incomplete data.
Researchers should also explain the assumptions underlying their primary analysis and present sensitivity analyses when alternative plausible assumptions could materially change the results.
Describing a dataset as complete after imputation is insufficient. Readers need to understand which values were observed, which were imputed, and how uncertainty was handled.
Transparent reporting is particularly important because the credibility of missing-data assumptions cannot be judged from a final regression table alone.
