How Can Differential Attrition Threaten Internal Validity?

What If the Participants Who Leave Your Study Are Different From Those Who Remain?

A researcher randomly assigns students to an intensive online learning program or conventional instruction. By the end of the semester, students who complete the online program achieve higher average scores.

However, several students assigned to the online program stopped participating because they struggled with the learning activities. Their final examination scores were not collected. Almost everyone in the comparison group completed the assessment.

The observed difference may therefore reflect not only the intervention but also who remained available for analysis.

This is the concern underlying differential attrition. When participant loss differs between research groups, especially when the missing outcomes are related to participants' likely performance, the groups represented in the final analysis may no longer provide a credible causal comparison.

How Does Differential Attrition Threaten Internal Validity?

How Participant Loss Can Distort Research Comparisons

What Is Attrition in Research?

Attrition refers to the loss of participants from some aspect of study participation or follow-up after they have entered a study.

Participants may withdraw consent, become unreachable, discontinue an intervention, transfer to another institution, miss follow-up assessments, or stop responding to researchers.

These circumstances are not necessarily equivalent. A participant who stops receiving an intervention may still provide outcome data. Another participant may complete the intervention but fail to provide the final outcome measurement.

For evaluating bias in an intervention effect estimate, a central concern is whether the outcome required for the intended analysis is observed.

Researchers should therefore distinguish treatment discontinuation, withdrawal from active participation, and missing outcome data rather than treating every departure as analytically identical.

What Makes Attrition Differential?

Differential attrition commonly refers to differences in the proportions of participants lost from intervention and comparison groups.

For example, an intervention group may lose 25% of participants while a comparison group loses 5%.

However, the methodological concern extends beyond unequal percentages. The processes producing missing outcomes may differ across groups even when their overall attrition rates are identical.

Suppose lower-performing students disproportionately leave the intervention group, while higher-performing students disproportionately leave the comparison group.

Both groups might lose the same percentage of participants, yet the observed outcome difference could be substantially distorted.

Consequently, researchers should examine both the amount of attrition and the relationship between missingness, treatment assignment, and potential outcomes.

Why Does Differential Attrition Threaten Causal Comparability?

A credible intervention comparison attempts to estimate what would happen under alternative treatment conditions for a defined population.

Random assignment can make treatment groups comparable with respect to baseline potential outcomes in expectation.

However, restricting the analysis to participants with observed follow-up outcomes may select different types of individuals from each randomized group.

Consider an intervention that is difficult for students with limited digital skills. If these students disproportionately fail to complete follow-up assessments in the intervention group, the observed intervention mean may be higher than the mean that would have been obtained from all assigned participants.

The resulting estimate may exaggerate the intervention's benefit.

Alternatively, participants who benefit most might leave because they no longer need support. Depending on the comparison group and missingness mechanism, the observed intervention effect could be underestimated.

Differential attrition is therefore a potential source of selection bias arising after assignment.

It is one of the internal validity threats that can undermine an otherwise credible comparison.

Why Unequal Attrition Rates Do Not Automatically Establish Bias

Unequal attrition percentages are useful diagnostic information, but they do not determine whether an effect estimate is biased.

Suppose more participants miss follow-up in one group because of an administrative scheduling error unrelated to their actual or potential outcomes.

Under suitable assumptions, the missingness may reduce precision without systematically distorting the outcome comparison.

By contrast, a relatively small amount of missing outcome data may cause substantial bias when missingness depends strongly on the unobserved outcome.

The Cochrane risk-of-bias framework therefore considers the proportion of missing outcomes together with the reasons for missingness and its possible relationship to true outcome values.

There is no universally valid percentage threshold below which attrition bias can automatically be dismissed.

Can Equal Attrition Rates Still Produce Selection Bias?

Yes. Equal percentages do not establish that the participants remaining in each group are comparable.

Imagine two groups that each lose 20% of participants.

In the intervention group, participants with the poorest outcomes are disproportionately missing. In the comparison group, participants with the strongest outcomes are disproportionately missing.

The observed difference between the groups may be exaggerated even though the attrition rates are identical.

Conversely, similar outcome-independent missingness processes may produce little bias despite unequal loss rates.

The essential question is not simply how many participants disappeared. It is whether the outcome data that remain provide a valid representation of the causal comparison.

How Do Missingness Mechanisms Affect Attrition Bias?

Missing-data methodology commonly distinguishes three mechanisms: missing completely at random, missing at random, and missing not at random.

These classifications concern the relationship between missingness and data values under a specified statistical framework.

These mechanisms should not be assigned merely by inspecting missing-data percentages.

For example, students who fail to submit a final assessment because of poor academic performance may have missing outcomes related to the values that would otherwise have been observed.

Whether an MAR assumption is defensible depends partly on what relevant predictors of missingness and outcomes have been measured.

The distinctions are important for evaluating missing-data threats to internal validity, but they do not replace a substantive investigation of why participants were lost.

How Is Differential Attrition Different From Ordinary Missing Data?

Missing data is a broad category encompassing absent values for outcomes, exposures, covariates, and other research variables.

Attrition specifically concerns participant loss or failure to remain under observation during follow-up.

A participant may remain enrolled but skip one questionnaire item, creating item-level missing data without necessarily being lost to follow-up.

Another participant may leave the study entirely, producing missing information across several later measurements.

These distinctions matter because prevention strategies, missingness mechanisms, and analytical options may differ.

For longitudinal research, investigators should document the timing and pattern of missing observations rather than reporting only a final retention percentage.

How Can Differential Attrition Interact With Selection Differences?

Groups may already differ at baseline because of nonrandom assignment. Attrition can then further alter their composition.

Suppose students who voluntarily enroll in an intensive intervention are initially more motivated than comparison students.

If less motivated intervention participants subsequently leave, the final observed intervention group may become even more selective.

The study may therefore face both baseline confounding and postassignment selection bias.

Researchers should distinguish selection differences that existed before intervention from changes in group composition arising during follow-up.

Both can threaten the intended causal interpretation, but they occur at different stages and may require different safeguards.

Does Randomization Prevent Attrition Bias?

No. Randomization protects baseline comparability in expectation, but it does not guarantee that every participant will provide outcome data.

If follow-up participation depends on treatment assignment and likely outcomes, analyzing only observed cases may undermine the randomized comparison.

For example, participants who experience an intervention as burdensome may be more likely to discontinue follow-up, especially when the burden is related to their outcome.

The final complete-case sample may then differ systematically from the original randomized population.

Researchers should preserve follow-up efforts across groups and avoid excluding participants merely because they discontinued the assigned intervention.

Does Intention-to-Treat Analysis Eliminate Differential Attrition?

Intention-to-treat analysis aims to estimate the effect of assignment to an intervention by analyzing participants according to their original randomized groups.

This principle helps preserve the interpretation of random assignment, particularly when participants do not adhere to their assigned conditions.

However, intention-to-treat analysis does not automatically resolve missing outcomes.

Researchers may retain participants' original group labels while still excluding those without follow-up measurements. Such an analysis remains vulnerable to missing-outcome bias.

Methods for handling missing outcomes require assumptions about the missing values and the processes generating missingness.

The distinction is important because nonadherence to an intervention and loss of outcome information are related but analytically different problems.

Can Multiple Imputation Correct Differential Attrition?

Multiple imputation replaces missing values with multiple sets of plausible values drawn from a specified statistical model. Estimates are then combined to reflect uncertainty associated with the missing information.

It can be useful when relevant predictors of outcomes and missingness have been measured and the required assumptions are defensible.

However, multiple imputation does not recover unobserved outcomes without assumptions. If participants leave because of outcome-related factors not adequately represented in the observed data, the imputation model may remain biased.

Similarly, inverse probability weighting can help under appropriate assumptions about the observation process, correct modeling, and adequate positivity.

No analytical method can guarantee the removal of attrition bias when essential information about missing outcomes is unavailable.

Sensitivity analyses are therefore important when plausible departures from the primary missingness assumptions could change the conclusions.

How Should Researchers Evaluate Attrition Rates?

At a minimum, report the number of participants assigned or enrolled, the number with observed outcomes, and the reasons for missingness where available.

Calculate attrition separately for each group and relevant follow-up point.

Some evidence-review frameworks use numerical attrition criteria for particular study types and evaluation purposes. Such criteria should be applied according to the relevant framework rather than treated as universal thresholds for internal validity.

How Can Researchers Reduce Attrition Before It Occurs?

Preventing avoidable missing outcomes is generally preferable to attempting to repair their consequences statistically.

Researchers can establish realistic follow-up schedules, use accessible assessment procedures, maintain appropriate contact arrangements, and reduce unnecessary participant burden.

In longitudinal educational research, obtaining permission to collect outcome information through institutional records may help when participants stop actively engaging with the intervention, provided ethical approvals and applicable privacy requirements permit such collection.

Retention procedures should be applied consistently across groups unless differences are justified by the design.

Researchers should also distinguish withdrawal from the intervention from withdrawal of consent for further data collection. A participant may stop using a learning platform while still agreeing to complete the final assessment.

Respect for participant autonomy remains essential. Retention strategies must not pressure participants to continue or override their right to withdraw.

How Do You Decide Whether Attrition Has Seriously Undermined the Study?

The assessment should consider the amount of missing outcome data, differences across groups, reasons for missingness, and plausible relationships between missingness and the true outcomes.

Researchers should also consider whether reasonable alternative assumptions about missing outcomes could materially change the estimated intervention effect.

A study with substantial attrition is not necessarily invalid, and a study with low attrition is not automatically unbiased.

The key question is whether the observed outcome comparison remains credible for the intended causal estimand.

How Selective Dropout Can Exaggerate the Apparent Benefit of an Intervention

The observed results are as follows.

The complete-case comparison suggests a seven-point advantage for the intervention group.

Now consider a hypothetical scenario in which the unobserved intervention students would have averaged 60, while the unobserved comparison students would have averaged 75.

The researchers cannot know the unobserved means merely from the complete-case dataset. The hypothetical values illustrate a sensitivity scenario, not a recovered result.

Other assumptions could produce different estimates, including a larger benefit, no effect, or a disadvantage.

The appropriate response is to investigate why outcomes were missing, use defensible analytical methods, and examine whether conclusions remain stable across scientifically plausible missing-outcome scenarios.

Common Misconceptions About Differential Attrition

How to Protect Internal Validity When Participants Leave Your Study

Plan follow-up procedures before recruitment begins. Identify the outcome information needed for the primary analysis and establish how it will be collected even when participants discontinue the intervention.

Record the timing and reasons for missing outcomes wherever feasible and ethically appropriate.

When evaluating an existing study, reconstruct participant flow separately for each comparison group and examine whether missingness could be related to the outcome.

Researchers should not treat statistical adjustment as a substitute for good follow-up procedures.

Whenever possible, prevent avoidable threats during study design by reducing participant burden, specifying retention procedures, and planning appropriate missing-data analyses.

When the remaining uncertainty is substantial, report it transparently and avoid presenting complete-case estimates as though they necessarily represent all originally assigned participants.

Differential Attrition Checks Before Reporting Study Results

Frequently Asked Questions About Differential Attrition

Who Remains in the Study Can Matter as Much as Who Entered It

How to Cite This Guide

This guide is intended to be read, shared, and used in research, teaching, and academic work. If you draw on its ideas, explanations, or other content, please acknowledge the source by citing the guide. Doing so gives appropriate credit and helps your readers locate the original resource.