03 · What You Need to Know
Balancing Comparable Follow-Up With Appropriate Participant Support
Why Retention Procedures Matter for Study Group Comparisons
Comparative research depends on obtaining information that supports a meaningful comparison between groups.
Random assignment can help establish comparable groups at baseline, but it does not guarantee that outcome data will remain equally available throughout follow-up.
If participants in one group are more likely to provide outcome measurements, the analyzed groups may differ from those originally assigned.
Whether this creates bias depends on the missing-data mechanism, the estimand, and the analytical assumptions.
For example, if participants experiencing poor outcomes disproportionately discontinue follow-up in one group, the observed outcomes may provide a misleading account of that group's results.
The National Research Council's report on missing data in clinical trials emphasizes preventing missing outcomes through study design and conduct. Retention procedures are therefore relevant to the validity of comparisons, not merely to administrative efficiency.
Equal Treatment Does Not Always Require Identical Procedures
Researchers should distinguish procedural uniformity from meaningful comparability.
Identical retention procedures
Every participant receives the same contact methods, reminder schedule, and follow-up arrangements regardless of circumstances.
Comparable retention opportunities
Participants have appropriately equivalent opportunities to provide required outcome information, with justified accommodations when necessary.
For example, one participant may prefer email while another requires accessible telephone communication. Providing these different options need not undermine comparability if the scientific measurement remains appropriate and the accommodations are applied consistently according to predefined rules.
Conversely, sending five reminders to intervention participants but only one to comparison participants may create unequal opportunities for outcome collection without a defensible reason.
The methodological objective is to avoid differences that systematically favor one group's outcome ascertainment.
Retention Contact Should Be Distinguished From Intervention Contact
Intervention groups often receive additional contact because of the treatment or activity being evaluated.
A digital learning intervention may involve weekly coaching sessions. A clinical intervention may require additional monitoring visits. A behavioral program may include regular facilitator communication.
These contacts are part of the intervention or its delivery and should not automatically be treated as retention activities.
Researchers should distinguish:
-
Intervention contact: Communication or interaction required to deliver the assigned intervention.
-
Outcome follow-up: Contact intended to obtain measurements needed for the research comparison.
-
Retention support: Procedures that facilitate continued participation or outcome assessment, such as reminders and scheduling assistance.
Intervention contact may incidentally improve retention by strengthening relationships with staff or making assessment appointments easier to remember.
Researchers should recognize this possibility when designing follow-up for comparison participants.
However, adding equivalent intervention-like contact to the comparison group could alter the treatment contrast. A comparison condition should not be redesigned solely to make total contact time identical unless that choice is scientifically justified.
How Unequal Follow-Up Can Create Differential Attrition
Differential attrition occurs when the proportion of participants missing follow-up information differs across study groups.
For a simple final-outcome assessment, group-specific attrition can be summarized as follows.
This difference describes outcome availability. It does not establish that the intervention caused dropout or that the treatment-effect estimate is necessarily biased.
Even equal attrition percentages can conceal different missing-data mechanisms. For instance, missingness may be associated with poor outcomes in one group and scheduling difficulties in another.
Researchers should therefore examine both the extent and plausible causes of missingness.
Why Equal Attrition Rates Do Not Guarantee Unbiased Results
Suppose both study groups retain 90% of participants at final follow-up.
In the intervention group, participants experiencing little improvement may be disproportionately missing. In the comparison group, missingness may be largely associated with temporary scheduling difficulties.
The groups have identical numerical retention rates but potentially different relationships between missingness and outcomes.
Conversely, unequal retention rates do not automatically establish a biased treatment comparison if the missing-data assumptions required by the analysis are credible.
Researchers should avoid interpreting the difference between attrition percentages as a complete diagnostic test of internal validity.
The implications depend on the target estimand, observed information, and assumptions about unobserved outcomes.
Standardize the Essential Outcome Assessment Procedures
Where possible, participants across groups should receive comparable information about when outcomes will be measured, what assessments involve, and how they can complete them.
Consistency is especially important when the measurement process itself could influence the outcome.
For example, if intervention participants complete an engagement questionnaire with a familiar facilitator while comparison participants complete it anonymously online, observed differences may reflect both group assignment and measurement conditions.
Researchers should consider whether assessment modes, instructions, timing, and outcome assessors can be standardized.
Blinded outcome assessment may be useful when feasible, particularly for outcomes susceptible to assessor expectations.
Where measurement procedures necessarily differ, the implications should be documented and evaluated.
Accessibility Accommodations Can Be Appropriate Across Groups
Providing identical procedures can sometimes create unequal practical opportunities to participate.
A participant with limited mobility may require a remote assessment. Another may need an accessible questionnaire format. Someone working irregular shifts may need an alternative appointment time.
Reasonable accommodations can improve access without necessarily compromising the research comparison.
Ideally, eligibility for accommodations should follow predefined, participant-centered criteria rather than discretionary decisions based on treatment assignment or expected outcomes.
Researchers should also consider whether the alternative assessment method measures the same construct comparably.
Accessibility is not a methodological inconvenience to be eliminated. It is a design consideration that should be addressed without introducing avoidable measurement differences.
Should Reminder Frequency Be Identical?
A common approach is to establish the same routine reminder sequence for all study groups.
This can reduce unexplained differences in opportunities to complete assessments and simplify implementation.
However, individual communication preferences or justified accessibility needs may require variation.
Researchers should distinguish predefined accommodation rules from additional reminders selectively provided to participants whose outcomes appear particularly important or favorable.
The latter practice could introduce selection problems.
The appropriate frequency of participant follow-up contact depends on assessment windows, participant circumstances, and the approved protocol.
Identical reminder counts should not become an inflexible objective when a participant has completed the assessment, withdrawn, or requested no further communication.
Can Retention Incentives Differ Across Groups?
Differences in reimbursement or incentives may sometimes be justified by different participation requirements.
For example, an intervention group attending additional research visits may incur greater transportation expenses.
Reimbursing actual participation-related expenses can be distinguished from offering different rewards for completing the same outcome assessment.
Unequal incentives for equivalent follow-up activities may influence outcome availability and should be carefully justified.
Researchers should also consider whether compensation arrangements could affect participants' behavior, perceived obligations, or willingness to withdraw.
All arrangements should follow the approved protocol and applicable ethical requirements.
Participant Safety May Require Unequal Contact
Some studies require additional safety monitoring for particular interventions or participants.
For example, an experimental treatment may warrant closer adverse-event monitoring than a comparison condition.
Such differences may be scientifically and ethically necessary.
However, safety monitoring and outcome retention should remain conceptually distinct.
Researchers should examine whether additional safety contact also increases opportunities for outcome collection or changes participants' experiences.
When these effects are plausible, they should be considered in the design and interpretation rather than assuming that necessary safety procedures are methodologically irrelevant.
Retention Differences Should Be Planned Before Data Collection
Retention procedures should be described in the research protocol, including routine contacts, permitted reminders, assessment windows, and arrangements for missed visits.
If different procedures are planned across groups, the rationale should be explicit.
Researchers should identify whether differences arise from intervention delivery, participant safety, accessibility, or another justified requirement.
These arrangements should be aligned with retention strategies specified in the research protocol.
Material changes during the study may require additional approval and should be documented, particularly when they could influence outcome availability or the treatment comparison.
Do Not Respond to Differential Attrition With Unplanned Selective Contact
Suppose interim monitoring reveals lower retention in the intervention group.
The research team may reasonably investigate whether the group faces additional burden or practical barriers.
However, selectively increasing reminders without considering the protocol, participant welfare, and analytical implications can introduce new differences between groups.
Researchers should first examine whether the observed pattern reflects scheduling, accessibility, intervention experiences, or other plausible mechanisms.
Any corrective procedure should be ethically appropriate, documented, and approved when required.
It should not depend on knowledge of individual outcomes or expectations about which participants will produce favorable results.
Statistical Analysis Cannot Automatically Correct Unequal Retention
Missing-data methods may help address incomplete outcomes under specified assumptions.
For example, multiple imputation or inverse probability weighting may be appropriate in certain designs when the required assumptions and information are available.
However, these methods cannot guarantee unbiased estimates when missingness depends on unobserved outcomes or other unmeasured factors.
Researchers should distinguish preventing missing outcomes from modeling the consequences of missingness.
Where differential attrition is plausible, the statistical analysis plan should specify relevant sensitivity analyses.
Such analyses can evaluate the dependence of conclusions on assumptions but cannot establish that all bias has been removed.
When Are Different Retention Strategies Scientifically Justified?
Differences may be defensible when they address legitimate needs without unnecessarily compromising the intended comparison.
| Situation |
Potentially justified approach |
Methodological consideration |
| Different accessibility needs |
Provide approved alternative contact or assessment arrangements |
Evaluate measurement comparability |
| Intervention-specific safety requirements |
Conduct necessary safety monitoring |
Distinguish safety contact from outcome ascertainment |
| Different intervention visit schedules |
Maintain comparable independent outcome follow-up |
Avoid confounding assessment opportunities with intervention contact |
| Participant communication preferences |
Use approved preferred channels |
Apply accommodation rules consistently |
| Unexpected group-specific retention barriers |
Investigate and propose proportionate corrective procedures |
Document changes and obtain required approvals |
The key question is whether the differences serve a legitimate scientific or participant-centered purpose and whether their consequences can be understood.
Watch Out
Do not equate identical retention percentages with unbiased comparisons, or unequal percentages with proof of bias. Evaluate who is missing, why missingness may occur, and how the analysis depends on assumptions about unobserved outcomes.
