03 · What You Need to Know
Recognizing When Anticipated Missingness Becomes a Fundamental Design Problem
What makes a research design scientifically unacceptable?
A research design is scientifically inadequate when its procedures cannot reasonably generate evidence capable of addressing its stated objective under defensible assumptions.
This is different from a study that merely faces practical difficulties. Research frequently involves uncertainty, imperfect measurement, and some loss of follow-up. Those limitations do not automatically invalidate a design.
The concern becomes more fundamental when essential information is predictably unavailable, when the remaining observations cannot support the intended comparison, or when the conclusions would depend almost entirely on unverifiable assumptions.
For example, a study designed to estimate twelve-month learning retention cannot answer that question directly if it collects only immediate post-intervention scores.
Similarly, a study of long-term outcomes may be difficult to justify if most participants cannot be followed and no suitable alternative outcome source exists.
Scientific acceptability therefore depends on alignment between the research question, measurement opportunities, available data, and proposed analysis.
Why is anticipated missingness different from observed missingness?
Anticipated missingness concerns the information researchers expect to lose before collection begins. Observed missingness concerns what actually becomes unavailable during the study.
The distinction matters because anticipated problems can sometimes be prevented through design changes.
For example, if a university knows that graduating students will lose access to institutional accounts, researchers can establish approved alternative contact procedures or arrange authorized outcome ascertainment before graduation.
Once participants become unreachable, those options may be much more limited.
Anticipated missingness should therefore influence whether a study is feasible, not merely how its final dataset will be analyzed.
This is why missing-data prevention should begin during research design.
Does a high anticipated missingness percentage automatically make a design unacceptable?
No. The proportion of missing data is an important consideration, but it is not sufficient for determining scientific acceptability.
A longitudinal study with incomplete intermediate measurements may remain informative if the observed data support an appropriate analysis under credible assumptions.
By contrast, a study with a relatively small amount of missingness may be seriously compromised if the missing values concern rare, consequential outcomes or essential confounders.
Researchers should evaluate missingness in relation to the primary estimand, the distribution of observed information, and the likely observation mechanism.
Understanding how much missing data would make the primary research question difficult to answer helps distinguish numerical monitoring thresholds from substantive validity judgments.
When does missingness threaten identification rather than merely precision?
Precision concerns how uncertain an estimate is, given the statistical model and information available. Identification concerns whether the target quantity can be determined from the observed-data distribution under the stated assumptions.
Missingness may reduce precision by leaving fewer observed outcomes. Under suitable assumptions, increasing recruitment or using additional valid information may improve precision.
Identification problems are different. If missingness depends on unobserved outcomes in ways that cannot be credibly modeled or constrained, the observed data may not uniquely determine the intended effect.
For example, suppose students with very poor unobserved performance are disproportionately unavailable for follow-up. Without additional information or assumptions, the observed outcomes may not establish the performance distribution of the full target population.
Recruiting more students through the same observation process does not necessarily solve that identification problem.
Precision problem
Too little observed information produces an estimate with uncertainty that may be too large for the research purpose.
Identification problem
The available data and defensible assumptions do not determine the intended quantity, even if the observed sample is large.
Both can make a design inadequate, but they require different responses.
What warning signs suggest that a design may be infeasible?
Several conditions warrant careful examination before recruitment begins.
| Warning sign |
Why it matters |
Possible response |
| Primary outcomes are expected to be unavailable for many participants. |
The study may lack sufficient information about the intended population. |
Improve ascertainment, evaluate alternative sources, or reconsider follow-up arrangements. |
| Missingness may depend strongly on unobserved outcomes. |
Primary estimates may depend heavily on unverifiable assumptions. |
Consider stronger measurement procedures, alternative designs, and sensitivity analyses. |
| Essential baseline confounders cannot be measured. |
The intended causal effect may not be identifiable through ordinary adjustment. |
Improve baseline measurement or reconsider the identification strategy. |
| Some study groups may have very little observed outcome information. |
Group comparisons may become unstable or unsupported. |
Review group-specific collection procedures and recruitment feasibility. |
| The required assessment occurs after participants become inaccessible. |
The proposed measurement schedule may be operationally unrealistic. |
Develop authorized follow-up arrangements or reconsider the study timeline. |
| Backup sources measure different constructs. |
Substituting them may change the research question or introduce measurement bias. |
Retain the original outcome or formally redefine and justify the alternative. |
These warning signs do not independently establish that a study must be abandoned. They indicate where feasibility and identification require explicit evaluation.
Why can predictable differential attrition make a design especially problematic?
Attrition may differ between intervention groups because of treatment burden, participant characteristics, dissatisfaction, adverse experiences, or outcome-related processes.
In a randomized study, assignment protects baseline comparability in expectation. However, selecting only participants with observed outcomes can undermine that protection when observation depends on assignment and outcome-related characteristics.
For example, if struggling students in the intervention group are more likely to disappear from follow-up, the remaining intervention participants may appear more successful than the original assigned group.
The direction of bias is not predetermined. Other selection mechanisms may produce the opposite result.
Researchers should therefore examine anticipated differences in missing-data patterns between study groups rather than relying on an overall attrition estimate.
Can researchers simply increase recruitment to compensate?
Increasing recruitment may help when the primary problem is insufficient precision and the observation process is otherwise compatible with valid estimation.
For example, if the study needs 200 observed outcomes and anticipates 20% missingness, recruiting approximately 250 participants may provide the required number of observed outcomes on average under the stated assumptions.
However, increasing recruitment does not automatically correct informative missingness.
If students with poor outcomes are systematically less likely to be observed, recruiting more students through the same procedures may produce a larger but similarly selected dataset.
Researchers should distinguish recruitment inflation intended to preserve precision from design changes intended to address bias.
In complex studies, simulation may be necessary to evaluate the expected information under different missingness patterns.
Can missing-data methods rescue an otherwise infeasible design?
Statistical methods can use incomplete data under specified assumptions, but they cannot automatically compensate for a fundamentally inadequate measurement design.
Multiple imputation may be useful when observed variables provide information about missing values and the imputation assumptions are credible.
Inverse probability weighting may address observation processes under suitable models and positivity assumptions. Likelihood-based methods may use partially observed longitudinal outcomes under appropriate conditions.
However, these methods may perform poorly when essential predictors are unavailable, observation probabilities approach zero, or the results depend strongly on unobserved outcomes.
In particular, no conventional method can reconstruct an entire unobserved outcome distribution without information or assumptions that support doing so.
The presence of statistical software capable of performing multiple imputation should not be treated as evidence that the design is scientifically adequate.
When are sensitivity analyses insufficient to justify proceeding?
Sensitivity analyses can reveal how conclusions change under plausible alternative assumptions about missing values.
They are especially important when missing-not-at-random mechanisms cannot be excluded.
However, sensitivity analysis is not a substitute for obtaining essential data when feasible.
If nearly every plausible scenario produces a different substantive conclusion, the study may be unable to provide a sufficiently informative answer for its intended purpose.
Alternatively, sensitivity analyses may show that the conclusion remains similar across a scientifically credible range of assumptions, supporting a more cautious interpretation.
The relevant criterion is whether the study can contribute meaningful evidence, not whether every sensitivity analysis produces the same statistical significance decision.
How can feasibility studies and pilot data inform the decision?
Pilot studies can provide evidence about assessment completion, participant availability, instrument burden, technical reliability, and follow-up procedures.
For example, a pilot may reveal that students who graduate before the final assessment cannot be contacted through the planned institutional system.
This information can motivate a redesign before the full study begins.
However, pilot estimates may be imprecise, especially when the pilot is small or unrepresentative.
Researchers should use plausible ranges of observation rates and consider how different scenarios affect precision and bias.
Where appropriate, simulation can combine anticipated recruitment, outcome distributions, missingness mechanisms, and planned estimators to evaluate whether the design is likely to provide useful information.
Should the measurement schedule be changed when follow-up is unlikely?
Possibly. Moving assessments or introducing flexible windows may reduce avoidable missingness.
However, changing the timing can alter the outcome being measured.
A study of twelve-month retention cannot simply replace that outcome with a six-week assessment and claim to answer the same question.
Similarly, collecting an immediate post-intervention score instead of a delayed outcome may change the scientific interpretation.
Researchers should consider whether changing the measurement schedule can reduce missingness while preserving the intended construct and estimand.
When the outcome time point must change substantially, the research question and protocol should be revised explicitly.
Can backup outcome sources make an otherwise infeasible design acceptable?
They may, when the alternative source provides the intended outcome with adequate validity, coverage, and authorization.
For example, official examination records may preserve the same standardized score for students who no longer complete research questionnaires.
However, a course grade that combines attendance, assignments, and participation is not automatically equivalent to a standardized examination score.
Researchers should evaluate whether the backup source measures the same construct, at the same relevant time, using compatible definitions.
The use of backup sources for critical research outcomes should be planned before data collection whenever feasible.
Data access must also comply with consent, ethics approval, and applicable governance requirements.
When should the research question be revised instead of the analysis?
A revised question may be necessary when the original outcome, target population, or follow-up period cannot be observed adequately.
For example, if twelve-month outcomes are unavailable but six-month outcomes can be measured reliably, researchers may consider a six-month effectiveness question.
That change may be scientifically valuable, but it is not equivalent to answering the original twelve-month question.
Similarly, a study may shift from estimating a causal effect to describing observed associations when consequential confounding or selection cannot be addressed credibly.
Such revisions should be made transparently and, where required, approved through the relevant protocol or ethics procedures.
Researchers should avoid changing the research question after examining results and then presenting the revised objective as though it had been prespecified.
What ethical concerns arise when a study is predictably unable to answer its question?
Research participation involves time, effort, and sometimes inconvenience or risk. Researchers have an obligation to justify those demands through a scientifically credible design.
If a study is highly unlikely to generate interpretable evidence about its stated objective, the ethical justification for involving participants may be weakened.
However, uncertainty about missingness does not automatically make research unethical. Many valuable studies involve difficult populations, rare outcomes, and unavoidable follow-up challenges.
The appropriate ethical assessment considers scientific value, feasibility, participant burden, risks, and whether the protocol includes reasonable procedures to protect data quality.
For research involving human participants, relevant institutional ethics requirements and applicable ethical frameworks should guide these decisions.
The World Medical Association's Declaration of Helsinki emphasizes the importance of scientifically sound research design and appropriate ethical review in medical research involving human participants. Its applicability should not be assumed to extend identically to every nonmedical research setting.
How should researchers make the final feasibility decision?
The decision should integrate scientific, statistical, operational, and ethical considerations.
First, determine whether the primary estimand can be identified under credible assumptions. Next, evaluate whether the expected observed information supports adequate precision.
Then consider whether feasible changes to recruitment, measurement, follow-up, or data sources can improve the design without changing the research question unintentionally.
If the design remains incapable of supporting the intended conclusion, researchers should revise it, narrow the objective, or reconsider whether proceeding is justified.
The absence of a universal missingness threshold makes this judgment more demanding, but also more scientifically meaningful.
