How Much Missing Data Would Make Your Primary Research Question Difficult to Answer?

At What Point Does Missing Data Make Your Research Question Difficult to Answer?

Suppose you are conducting a semester-long study of a digital learning intervention. You recruited 250 students, but only 200 completed the final assessment. The remaining 50 students have missing primary outcomes.

That represents 20% missing outcome data. Is the percentage acceptable? Should you continue with the analysis, recruit additional participants, or conclude that the study can no longer answer its primary research question?

Researchers frequently encounter suggested thresholds such as 5%, 10%, or 20%. These figures can appear reassuringly precise, particularly when a dissertation committee or reviewer asks whether the amount of missing data is acceptable.

Yet the percentage alone cannot establish whether a study remains valid. Twenty missing outcomes among randomly selected participants may have different implications from twenty missing outcomes concentrated among participants experiencing the poorest results.

The more useful question is whether the information that remains, together with defensible assumptions about what is missing, can support the intended estimate with sufficient precision and credibility.

There Is No Universal Missing-Data Percentage That Determines Research Validity

How to Determine Whether Missing Data Threaten the Primary Research Question

Why is there no universally acceptable percentage of missing data?

The consequences of missingness depend on the research question and the process that generated the observed data.

A study with 3% missing primary outcomes may be substantially biased if those outcomes are concentrated among participants experiencing a particular adverse event. Another study with 25% missing repeated measurements may still provide useful estimates when extensive observed information supports an appropriate longitudinal analysis under credible assumptions.

Neither example establishes a general rule about acceptable missingness.

Methodological discussions sometimes use percentage thresholds as practical warnings. However, a warning threshold is not equivalent to a validated boundary between acceptable and unacceptable research.

The National Research Council's recommendations on missing data emphasize prevention, the importance of the estimand, and sensitivity to missingness assumptions rather than reliance on a universal allowable proportion.

Researchers should therefore distinguish an administrative threshold adopted for monitoring from a methodological conclusion about the validity of an estimate.

Which missing variables matter most?

Missingness in the primary outcome is often especially consequential because the outcome directly defines the principal research question.

However, missing exposure information can prevent correct classification of participants, while missing essential confounders can compromise causal adjustment. Missing cluster identifiers, event dates, or baseline measurements may also undermine particular estimators.

Missingness in a secondary descriptive variable may have less direct influence on the primary analysis, although it could remain important for another objective.

This distinction reinforces the importance of identifying variables at risk of missing data before collection begins.

Why does the missing-data mechanism matter more than the percentage alone?

Missing completely at random (MCAR), missing at random (MAR), and missing not at random (MNAR) describe different relationships between missingness and observed or unobserved information.

Under MCAR, missingness is independent of the relevant data values. Under MAR, the probability of missingness does not additionally depend on missing values after conditioning on appropriate observed information. Under MNAR, dependence on unobserved information remains.

Consider students who fail to complete a final examination. If missingness results from an independently occurring technical failure, an MCAR mechanism may be plausible. If completion depends on recorded baseline achievement and attendance, MAR may be plausible conditional on those variables.

If students with particularly poor unobserved examination performance are less likely to complete the assessment even after accounting for available information, MNAR may be a concern.

These mechanisms generally cannot be distinguished conclusively using observed data alone.

Importantly, missingness under MAR is not automatically harmless. Appropriate estimation and modeling assumptions are still required. Similarly, MNAR does not mean that every analysis is impossible, but it introduces assumptions that cannot generally be identified from the observed data without additional information.

How can a small amount of missing data produce substantial bias?

Suppose a study examines a rare adverse outcome. If most of the missing observations concern participants who experienced that event, even a relatively small overall missingness percentage may materially distort the estimated event frequency.

For example, missing five events from a population in which only fifteen events occurred could be consequential, even when the total study sample is large.

Similarly, missingness concentrated among participants with extreme outcome values can change means, proportions, or group comparisons disproportionately.

The concern is not merely how many observations are missing, but which observations are unavailable and how their values might differ from those observed.

Researchers should avoid assuming that a small missingness percentage guarantees negligible bias.

How does missingness reduce statistical precision?

Missing outcomes generally reduce the amount of information available for estimating the primary effect, although the consequences depend on the estimator and data structure.

In a simple comparison of two independent groups, losing observations may increase standard errors and widen confidence intervals.

However, the reduction in precision is not necessarily proportional to the percentage missing. Missingness concentrated in one group, one cluster, or one important subgroup may have greater consequences than the same overall proportion distributed more evenly.

Longitudinal analyses may retain information from participants with incomplete follow-up, while methods using auxiliary variables can sometimes improve efficiency under suitable assumptions.

The relevant question is whether the resulting uncertainty remains acceptable for the decision the study is intended to inform.

How should sample size calculations account for missing outcomes?

Researchers sometimes inflate recruitment targets to compensate for anticipated missingness.

For a simple illustration, suppose the study requires 200 observed primary outcomes and expects 20% of participants to have missing outcomes.

Recruiting 250 participants would yield approximately 200 observed outcomes on average under the assumed observation rate.

However, the calculation does not correct selection bias or guarantee adequate precision. It also does not account for complex missingness patterns, clustering, or the information retained through longitudinal models.

When the primary analysis is complex, simulation may provide a more defensible assessment than simple recruitment inflation.

Why should missingness be examined separately across study groups?

Overall missingness can conceal substantial differences between exposure or intervention groups.

Suppose 10% of outcomes are missing in one group and 30% in another. This difference may reflect intervention burden, participant characteristics, measurement procedures, or outcome-related nonresponse.

Unequal rates do not automatically establish bias. Nevertheless, they raise questions about whether the observed groups remain comparable for the intended estimand.

Equal rates are not a guarantee either. Different types of participants may be missing in each group despite identical percentages.

Researchers should examine missing-data patterns between study groups, including relevant baseline characteristics and follow-up occasions.

Can complete-case analysis still be valid?

Yes, under particular assumptions. Complete-case analysis is not universally biased, and its validity does not require MCAR in every setting.

However, excluding incomplete observations can reduce precision and may create selection bias when the complete cases do not support the intended comparison.

For example, if completion depends on intervention assignment and outcome-related characteristics, conditioning on complete cases can introduce a noncausal association.

The appropriateness of complete-case analysis depends on the missingness process, the estimand, and the statistical model.

Researchers should not select it solely because the missingness percentage falls below an arbitrary cutoff.

Can multiple imputation make high missingness acceptable?

Multiple imputation can use observed information to represent uncertainty about missing values across multiple completed datasets.

Under suitable assumptions and an appropriate imputation model, it may improve estimation compared with discarding incomplete records.

However, imputation does not recreate the unobserved values as facts. Its validity depends on the information available, the missingness assumptions, and the compatibility of the imputation and analysis models.

As missingness becomes more extensive, conclusions may become increasingly dependent on assumptions about unobserved outcomes.

There is no percentage above which multiple imputation automatically fails, nor a percentage below which it automatically succeeds.

Sterne and colleagues emphasize that multiple imputation has important applications but can be misleading when used without attention to its assumptions and limitations.

How can sensitivity analysis help determine whether the question remains answerable?

Sensitivity analysis examines how conclusions change under alternative plausible assumptions about missing values.

For example, researchers may examine scenarios in which unobserved examination scores are systematically lower than scores predicted under a MAR-based analysis.

Pattern-mixture models, selection models, and other approaches can support such investigations when appropriate to the data and estimand.

A tipping-point analysis may identify the assumptions under which the substantive conclusion changes.

These methods do not reveal the true unobserved outcomes. They clarify how strongly the conclusion depends on assumptions that the available data cannot fully verify.

If plausible alternative assumptions produce materially different conclusions, the study may not provide a sufficiently robust answer to its primary question.

Can partial identification be more informative than assuming missing outcomes?

Sometimes. When missing outcomes cannot be credibly modeled, researchers may report ranges of possible estimates under explicitly stated restrictions.

For a binary outcome, simple worst-case bounds can be constructed by considering all missing outcomes as events and then as nonevents.

These bounds may be wide, especially with substantial missingness, but they avoid claiming to know values that were never observed.

Additional credible assumptions may narrow the bounds. However, those assumptions should be stated rather than hidden within the analysis.

Partial identification is particularly useful when the observed data cannot support a precise point estimate without strong unverifiable assumptions.

When does missingness make the primary research question difficult to answer?

The difficulty becomes substantial when the remaining observations provide inadequate precision, important comparison groups have insufficient information, or plausible missingness mechanisms produce materially different estimates.

For causal studies, missingness may also undermine the identification assumptions needed to interpret the estimate as an effect.

A study may remain capable of describing observed participants while no longer supporting the originally intended population-level or causal conclusion.

Researchers should distinguish those outcomes rather than treating the entire project as either completely valid or scientifically worthless.

The decision should be based on the estimand, available information, uncertainty, and sensitivity to assumptions.

What should be planned before data collection?

Researchers should identify essential variables, anticipated observation rates, vulnerable subgroups, and feasible procedures for preventing or recovering missing information.

They should also specify the primary missing-data approach and sensitivity analyses, including the assumptions each method requires.

Monitoring thresholds may be established to trigger operational review, but these should not be confused with universal validity thresholds.

For example, a protocol may require the research team to investigate when more than 10% of scheduled outcomes are missing. That is a project-specific management rule, not evidence that 9% missingness is harmless or 11% invalidates the study.

These decisions should be integrated into missing-data planning during research design.

Evaluating Whether Missing Outcomes Could Change a Study's Conclusion

The observed-only difference is 10 percentage points in favor of tutoring. However, the complete-group pass proportions depend on the 40 unobserved outcomes.

These bounds are intentionally broad because they make no substantive assumptions about the missing outcomes beyond the binary outcome definition.

The example does not establish that 20% missingness is unacceptable. It demonstrates why the proportion alone cannot determine whether the primary conclusion is secure.

With stronger evidence about the observation process, the study may support narrower estimates. Without it, uncertainty about the unobserved outcomes remains consequential.

Common Misconceptions About Acceptable Missing-Data Percentages

Evaluate Whether Missingness Changes the Answer, Not Just the Sample Size

Begin with the primary estimand and identify which measurements are necessary to estimate it.

Examine missingness by variable, group, and assessment occasion. Then assess the resulting precision and the plausibility of the assumptions needed to interpret the remaining observations.

Where possible, investigate documented reasons for missing data and whether valid backup outcome sources can recover information.

The objective is not to defend a particular percentage. It is to demonstrate that the available evidence can support the intended conclusion under assumptions that are sufficiently credible and transparent.

Before Deciding Whether Missingness Is Too Extensive

Frequently Asked Questions About Missing-Data Thresholds

The Consequences of Missing Data Matter More Than the Percentage

How to Cite This Guide

This guide is intended to be read, shared, and used in research, teaching, and academic work. If you draw on its ideas, explanations, or other content, please acknowledge the source by citing the guide. Doing so gives appropriate credit and helps your readers locate the original resource.