Is Randomization Enough to Support Causal Claims When Treatment Adherence Is Poor?

What Happens to a Randomized Experiment When Participants Do Not Follow Their Assigned Treatment?

You are conducting a randomized evaluation of an AI-supported formative feedback system. Students are assigned to either AI-supported feedback or conventional teacher feedback.

The allocation procedure is properly implemented. However, the intervention does not proceed as planned.

Some students assigned to AI-supported feedback rarely access the system. Others discontinue using it after several weeks. Meanwhile, some students assigned to conventional feedback independently use similar generative AI tools.

At the end of the semester, you observe a modest difference in writing performance between the randomized groups.

Can the result still support a causal claim? Should you exclude students who did not follow their assigned condition? Would comparing actual AI users against nonusers provide a more accurate estimate of effectiveness?

These questions require distinguishing the effect of being assigned an intervention from the effect of actually receiving or adhering to it. Randomization is powerful, but its causal contribution depends on the effect the researcher intends to estimate.

Randomization Can Support an Assignment Effect, but Not Automatically an Adherence Effect

Understanding Randomization, Treatment Adherence, and Causal Effects

What Does Randomization Actually Establish?

Randomization assigns eligible participants to intervention conditions using a chance mechanism.

When implemented appropriately, it supports comparability of assigned groups in expectation with respect to baseline potential outcomes.

For example, students may be randomly assigned access to an AI-supported feedback system or conventional feedback.

However, randomization determines assignment, not necessarily the treatment participants subsequently receive.

Students may decline the intervention, use it inconsistently, switch conditions, or obtain similar support outside the study.

These behaviors can affect the relationship between randomized assignment and actual exposure.

Researchers should therefore distinguish the causal effect of assignment from effects defined by treatment receipt or adherence.

What Is Treatment Adherence?

Treatment adherence concerns the extent to which participants follow the intervention strategy or protocol relevant to the study.

In educational technology research, adherence may involve accessing a platform, completing specified activities, using feedback as instructed, or remaining within the assigned learning condition.

However, adherence must be defined according to the intervention.

For example, logging into an AI feedback system is not necessarily equivalent to engaging meaningfully with its suggestions.

Likewise, an adaptive intervention may intentionally allow variation in how participants use the system.

Researchers should distinguish protocol-defined adherence from differences in engagement that are legitimate features of the intervention.

What Types of Nonadherence Can Occur?

These events may have different causal and practical implications. Researchers should document them rather than combine all deviations into one vague measure of noncompliance.

What Is the Intention-to-Treat Principle?

The intention-to-treat principle generally involves analyzing participants according to their original randomized assignment, regardless of subsequent adherence.

For example, students assigned AI-supported feedback remain in the intervention group for the assignment-based comparison even if they rarely use the system.

This preserves the baseline comparability generated by randomization in expectation.

Under appropriate conditions, the analysis estimates the causal effect of assignment to the intervention strategy.

However, intention-to-treat should not be interpreted as a guarantee that all biases are absent. Missing outcomes, interference, and other problems can still threaten validity.

Gupta (2011) discusses the intention-to-treat concept and its role in randomized controlled trials.

What Is the Difference Between Assignment and Treatment Receipt?

Assignment concerns the condition allocated through randomization. Treatment receipt concerns the intervention actually experienced.

Suppose a university randomly assigns students to receive access to an AI feedback system.

The effect of offering access may be important for institutional policy because it reflects what happens when the university implements that strategy under realistic participation conditions.

However, it differs from the effect of sustained use among students who actually engage with the system.

The first is an assignment or treatment-policy question. The second concerns a treatment-receipt or adherence-related causal effect.

Researchers should specify which question the study is intended to answer before selecting an analysis.

What Is the Intention-to-Treat Effect?

For a simple randomized comparison, the intention-to-treat effect can be expressed as:

If adherence is poor, the assignment effect may be smaller than the effect of actually receiving the intervention under full adherence.

However, that relationship is not guaranteed. Nonadherence may interact with treatment effects, implementation conditions, or other processes.

A small intention-to-treat estimate should therefore not automatically be interpreted as evidence that the intervention itself is ineffective when used as intended.

Equally, researchers should not assume that perfect adherence would necessarily produce a larger beneficial effect.

Does Poor Adherence Destroy Randomization?

Not necessarily. The original randomized assignment remains a randomized variable.

If participants are analyzed according to their assigned groups, the assignment contrast can retain the advantages of randomization under appropriate conditions.

However, restricting analysis to adherent participants changes the comparison.

Adherence may be related to motivation, prior achievement, intervention experiences, or emerging outcomes.

Consequently, the adherent subset may no longer preserve the comparability established by original randomization.

Researchers should distinguish the integrity of random assignment from the comparability of groups defined by post-randomization behavior.

Why Can Excluding Nonadherent Participants Introduce Bias?

Nonadherence often occurs after randomization and may be influenced by characteristics related to outcomes.

For example, highly motivated students may be more likely to complete all AI feedback activities and achieve higher writing scores.

If researchers compare only adherent intervention participants with adherent comparison participants, the groups may differ systematically in motivation or other prognostic characteristics.

Randomization does not guarantee exchangeability within subsets defined by post-randomization adherence.

Excluding nonadherent participants may therefore introduce selection bias.

Researchers should not assume that removing protocol deviations produces a purer causal comparison.

What Is a Per-Protocol Effect?

A per-protocol effect concerns outcomes under specified intervention strategies when adherence to the relevant protocol is maintained.

For example, researchers may want to estimate the effect of consistently using AI-supported feedback throughout a semester compared with consistently receiving conventional feedback.

This is a different causal question from the effect of assignment.

However, simply analyzing participants who adhered to their assigned condition does not automatically identify the per-protocol effect.

Adherence may depend on time-varying characteristics that also influence outcomes.

Hernán and Robins (2017) discuss the distinction between intention-to-treat and per-protocol effects and the analytical challenges associated with estimating the latter.

What Is an As-Treated Analysis?

An as-treated analysis classifies participants according to the treatment they actually received rather than their original randomized assignment.

For example, students may be analyzed as AI feedback users or nonusers regardless of their assigned condition.

This may appear to provide a more direct comparison of treatment receipt.

However, actual treatment use may be self-selected and associated with characteristics that influence outcomes.

Consequently, an ordinary as-treated comparison may lose the causal protection provided by randomization.

Researchers should not assume that classifying participants according to actual exposure automatically estimates the causal effect of treatment receipt.

How Do the Main Analytical Approaches Differ?

The appropriate analysis depends on the estimand. Researchers should not select whichever approach produces the largest treatment difference.

Can Randomization Be Used as an Instrument for Treatment Receipt?

Potentially. Randomized assignment may serve as an instrumental variable for actual treatment receipt under additional assumptions.

For example, assignment to AI feedback access may influence whether students use the system.

Under conditions including instrument relevance, an appropriate exclusion restriction, and other assumptions, instrumental-variable methods may identify a causal effect for a particular population.

However, randomization alone does not guarantee that the exclusion restriction holds.

Assignment may influence outcomes through pathways other than the measured treatment receipt, such as expectations, additional teacher attention, or access to related resources.

Researchers should justify the instrumental-variable assumptions rather than assume that randomized assignment automatically provides a valid instrument for every adherence question.

What Is the Complier Average Causal Effect?

The complier average causal effect, or CACE, concerns a causal effect among participants whose treatment receipt would follow their randomized assignment under the relevant definition of compliance.

For a simple binary assignment and treatment setting, the Wald estimator may be written as:

Angrist, Imbens, and Rubin (1996) provide a foundational treatment of causal effects using instrumental variables and compliance types.

The complier effect is not necessarily the average treatment effect for every participant.

Its interpretation depends on the compliance definition, population, and assumptions.

What Is the Exclusion Restriction?

In an instrumental-variable analysis, the exclusion restriction requires that the instrument affect the outcome only through the treatment pathway specified by the model.

For example, assignment to AI feedback access should influence writing performance only through the relevant treatment receipt if the exclusion restriction is to hold in that formulation.

However, assignment may also change student expectations, teacher behavior, or access to other learning activities.

Such pathways could violate the exclusion restriction.

The assumption is substantive and cannot generally be verified solely from the observed data.

Researchers should examine whether the intervention and implementation setting make it plausible.

What Is Monotonicity?

Monotonicity is an assumption used in common instrumental-variable interpretations of compliance.

Informally, it rules out participants who would take the treatment when assigned to the comparison condition but would not take it when assigned to the intervention condition, under the relevant binary treatment definition.

In educational settings, this assumption may be difficult to justify when students can independently access similar technologies.

For example, some students might use an AI tool only when not assigned the institution's official system.

Researchers should not assume monotonicity merely because assignment was randomized.

The plausibility of the assumption depends on how treatment receipt is defined and how students respond to assignment.

What Happens When Control Participants Use the Intervention?

Contamination occurs when participants assigned to the comparison condition receive or access aspects of the focal intervention.

For example, students assigned conventional feedback may independently use generative AI tools outside the university's intervention platform.

This can reduce the practical difference between assigned conditions.

However, contamination does not necessarily invalidate the effect of assignment as a policy-relevant estimand.

The result may reflect the effect of offering the institutional intervention in an environment where alternative AI tools are already available.

Researchers should document contamination and clarify whether the intended question concerns assignment, access, or actual use.

Can Poor Adherence Reduce Statistical Power?

Yes. When assignment produces only a modest difference in actual treatment exposure, the outcome contrast between assigned groups may be smaller than under stronger adherence.

This can reduce the ability to detect a specified assignment effect under conventional hypothesis testing.

However, the magnitude and direction of the relationship depend on the intervention and adherence processes.

Researchers should not assume that low adherence necessarily explains every nonsignificant result.

Low statistical power, implementation failure, and a genuinely small intervention effect are different possibilities.

Confidence intervals and process evidence may help clarify the uncertainty, but do not automatically distinguish all explanations.

Does Poor Adherence Mean the Intervention Failed?

Not necessarily. Poor adherence may reflect implementation barriers, participant burden, technical difficulties, limited perceived usefulness, or a mismatch between the intervention and the educational setting.

These findings may be important in their own right.

For example, an AI feedback system that improves writing under intensive supervised use may still produce little benefit when offered under ordinary institutional conditions because students rarely engage with it.

The difference between efficacy under specified adherence and effectiveness under realistic implementation is relevant to educational policy.

Researchers should distinguish the intervention's potential effect under sustained use from the effect of implementing an access or assignment strategy.

Both questions may be scientifically important, but they require different estimands and evidence.

How Should Adherence Be Measured?

Adherence measurement should reflect the intervention's essential components.

For example, a study may record platform access, completion of feedback activities, duration of use, or documented application of feedback to revisions.

However, no single indicator necessarily captures meaningful engagement.

A student may log into the system frequently without applying its suggestions, while another may use it less often but engage deeply with feedback.

Researchers should define adherence prospectively and evaluate the validity of the chosen indicators.

Thresholds distinguishing adherent and nonadherent participants should be substantively justified rather than selected after examining outcomes.

Why Does the Timing of Adherence Matter?

Adherence may change throughout the intervention period.

Students may begin with high engagement and gradually discontinue use, or they may increase use after receiving poor assessment feedback.

These changes may be related to evolving academic confidence, workload, or performance.

Consequently, adherence can be influenced by time-varying characteristics that also affect outcomes.

Estimating the effect of sustained adherence may require methods that account for treatment-confounder feedback.

Hernán and Robins (2017) discuss why appropriately adjusted per-protocol analyses may require causal methods beyond simply excluding nonadherent participants.

How Does Missing Outcome Data Differ From Nonadherence?

Nonadherence concerns whether participants follow the intervention strategy. Missing outcome data concern whether the outcomes needed for analysis are observed.

A participant may discontinue using AI feedback but still complete the final writing assessment.

Conversely, a participant may adhere fully but fail to provide the final outcome.

These situations have different analytical implications.

Intention-to-treat analysis does not automatically resolve bias caused by missing outcomes.

Researchers should document both adherence and outcome ascertainment and consider appropriate methods for each problem.

Can Researchers Improve Adherence During the Trial?

Potentially, but changes should be consistent with the trial protocol and intended intervention strategy.

Researchers may provide technical support, reminders, or other planned implementation assistance.

However, introducing new incentives or instructional activities after observing poor adherence may change the intervention being evaluated.

Such changes should be documented and assessed for their implications for the estimand and trial integrity.

Researchers should distinguish ordinary implementation support from substantive modifications to the intervention.

Where possible, adherence risks should be anticipated during pilot testing and intervention planning.

Should the Primary Analysis Always Be Intention-to-Treat?

Intention-to-treat is often appropriate when the central question concerns the effect of assignment or offering an intervention.

However, the appropriate primary analysis depends on the trial objective and estimand.

Some research questions specifically concern outcomes under sustained adherence, while others concern the effectiveness of an implementation policy.

Researchers should prespecify the target effect and choose methods consistent with it.

The ICH E9(R1) addendum emphasizes defining estimands and strategies for handling events that occur after treatment begins.

Although developed for clinical trials, this principle is relevant to educational experiments involving nonadherence, treatment switching, or intervention discontinuation.

How Should Researchers Interpret a Small Intention-to-Treat Effect?

A small assignment effect may indicate that offering the intervention produces limited average benefit under the studied conditions.

However, it does not necessarily establish that the intervention would have little effect under sustained use.

Likewise, a small assignment effect should not automatically be dismissed as dilution caused by poor adherence.

Adherence may be associated with other factors, and the intervention may have heterogeneous effects.

Researchers should interpret the assignment effect according to its estimand and use appropriate additional analyses when other causal questions are important.

What If Adherence Is Related to the Outcome?

This is a central concern for adherence-based comparisons.

Students who expect to perform well may engage more actively with feedback. Students experiencing difficulties may discontinue use or seek additional support.

As a result, adherence and outcome may share common causes.

Comparing adherent and nonadherent participants may therefore produce a biased estimate of the treatment effect.

Researchers should identify plausible adherence-outcome confounders and consider whether suitable causal methods and measurements are available.

Randomization of initial assignment does not remove confounding introduced by subsequent self-selected adherence.

How Do You Design a Trial to Address Expected Nonadherence?

Researchers should anticipate adherence problems during study planning rather than treat them only as analytical complications.

Important considerations include intervention acceptability, technical access, participant burden, contamination risks, adherence measurement, and the distinction between assignment and receipt effects.

The protocol should specify the primary estimand, planned analysis, and handling of relevant post-randomization events.

Where adherence-based effects are important, researchers should determine whether the necessary confounders and exposure histories can be measured.

Sample-size planning should also consider the expected assignment contrast and realistic implementation conditions.

These choices should be aligned with the broader features that strengthen causal interpretation.

How Should Poor Adherence Be Reported?

Researchers should report the randomized allocation, actual intervention receipt, adherence definitions, deviations, contamination, and missing outcome data.

The primary analysis should be identified according to its estimand.

Additional per-protocol or as-treated analyses should be described with their assumptions and limitations.

Researchers should avoid presenting an unadjusted adherent-only comparison as though it retained the causal protection of randomization.

The discussion should distinguish evidence about the effectiveness of assignment from evidence concerning treatment receipt or sustained use.

Transparency about adherence helps readers understand what intervention was actually evaluated and what conclusions the trial supports.

Evaluating AI-Supported Feedback When Many Students Do Not Use It

Suppose the intention-to-treat estimate indicates a modest improvement in writing performance.

This may support a causal interpretation of the effect of assignment to AI-supported feedback under the trial's conditions.

However, comparing the 90 adherent intervention students against all comparison students would not automatically establish the effect of sustained AI feedback use.

The adherent students may differ in motivation, prior ability, or other characteristics associated with writing performance.

The trial can therefore retain a meaningful randomized causal comparison despite poor adherence, while the effect of sustained use remains a separate question requiring additional assumptions.

Common Misconceptions About Randomization and Treatment Adherence

How to Interpret Randomized Trials With Poor Treatment Adherence

Begin by identifying whether the research question concerns assignment to the intervention or actual receipt and sustained adherence.

Then determine which analysis corresponds to the intended effect and what additional assumptions it requires.

Poor adherence does not necessarily eliminate causal inference, but it makes precise definition of the causal question especially important.

Check Whether Your Randomized Trial Supports the Intended Causal Claim

Frequently Asked Questions About Randomization and Poor Adherence

Randomization Protects the Assignment Comparison, Not Every Post-Randomization Comparison

How to Cite This Guide

This guide is intended to be read, shared, and used in research, teaching, and academic work. If you draw on its ideas, explanations, or other content, please acknowledge the source by citing the guide. Doing so gives appropriate credit and helps your readers locate the original resource.