03 · What You Need to Know
How Threats to Internal Validity Affect Research Conclusions
Internal Validity Concerns the Credibility of a Causal Comparison
Internal validity concerns whether the design and execution of a study support a causal interpretation of its findings. In an intervention study, the central issue is whether the intervention produced the observed outcome difference.
Imagine comparing two possible outcomes for the same participants: what would happen if they received an intervention and what would happen if they did not? This is the fundamental problem of causal inference. Researchers cannot ordinarily observe both potential outcomes for the same person under the same circumstances.
Instead, they construct comparisons using control groups, baseline measurements, repeated observations, or other research designs. Internal validity depends on how convincingly these comparisons approximate the relevant counterfactual outcome.
A threat becomes consequential when it offers a plausible alternative explanation for the estimated effect.
Selection Differences Can Make Groups Incomparable Before an Intervention Begins
Selection threatens internal validity when participants in comparison groups differ in characteristics that also influence the outcome.
For example, students who voluntarily enroll in an intensive tutoring program may be more motivated than students who do not enroll. If motivation affects academic achievement, comparing their final scores may attribute preexisting differences to tutoring.
These differences may involve measured characteristics, such as prior achievement, or unmeasured characteristics, such as persistence or family support.
Random assignment can reduce systematic baseline differences in expectation, although chance imbalances and implementation problems remain possible. In nonrandomized studies, researchers must establish why the groups are sufficiently comparable for the intended causal interpretation.
The specific mechanisms and safeguards are examined in selection differences between research groups.
History Effects Can Produce Changes Unrelated to the Intervention
A history effect occurs when an event outside the intervention influences the outcome during the study period.
Suppose a university introduces an online learning platform and finds that students' performance improves afterward. During the same semester, the university also implements a new academic support program.
The improvement might reflect the platform, the support program, or their combined influence.
History is particularly problematic in single-group before-and-after studies because the passage of time permits other events to affect outcomes. A concurrent comparison group may help, provided the external events affect the groups similarly or their differential effects can be addressed.
Researchers using repeated measurements should consider whether events occurring between baseline and follow-up could explain the apparent change.
Maturation Can Produce Improvement or Decline Without Treatment
Maturation refers to changes that arise through ordinary developmental, biological, psychological, or other time-related processes rather than through the intervention itself.
Children may improve their reading ability as they grow older. Patients may recover naturally. Students may become more proficient through accumulated experience.
If a study measures participants before and after an intervention without an appropriate comparison, these changes may be mistaken for treatment effects.
Maturation does not necessarily mean physical aging. It can include fatigue, adaptation, increasing familiarity with a task, or other systematic changes over time.
The central question is whether the observed change exceeds what would reasonably have occurred without the intervention.
Regression to the Mean Can Make Extreme Scores Appear to Improve Naturally
Regression to the mean is a statistical phenomenon in which unusually extreme observations tend, on average, to be less extreme on subsequent measurements when the measurements are imperfectly correlated.
Suppose a school selects students for an intervention because they scored exceptionally poorly on a diagnostic assessment. Some students may score higher on a second assessment even without effective intervention, partly because their initial scores reflected temporary circumstances or random measurement variation.
Without an appropriate comparison, this movement toward more typical scores may be interpreted as improvement caused by the program.
The risk is especially relevant when eligibility depends on unusually high or low baseline measurements. Understanding regression to the mean helps researchers avoid treating every rebound as evidence of effectiveness.
Attrition Can Change Who Remains in the Comparison
Attrition occurs when participants withdraw, become unreachable, or otherwise stop contributing outcome information.
Attrition becomes particularly concerning when the probability of remaining in the study is related to treatment assignment and potential outcomes.
For example, if students who struggle with a digital intervention are more likely to leave the intervention group, the final average may disproportionately represent students who benefited or found the platform easier to use.
Even similar overall attrition rates across groups do not guarantee the absence of bias. The reasons for missing outcomes and the characteristics of those who leave also matter.
Researchers should examine how attrition changes the composition of comparison groups rather than relying exclusively on the percentage of participants retained.
Measurement Changes Can Create Artificial Differences
A comparison may be distorted when measurement procedures differ across groups or change over time.
Examples include using a more difficult examination at follow-up, changing the scoring rubric midway through a study, replacing a measurement device, or allowing assessors to apply different standards across conditions.
These problems are sometimes described as instrumentation threats. Their effect depends on whether the change introduces systematic differences into the outcome being compared.
A new assessment is not automatically invalid. The concern is whether scores remain meaningfully comparable after the change.
Researchers should document measurement procedures and examine whether changes in instruments or assessment practices could account for the observed results.
Contamination and Nonadherence Can Complicate Treatment Comparisons
Contamination occurs when participants assigned to one condition receive elements of another condition. For example, students in a control group may obtain access to learning materials intended exclusively for the intervention group.
Nonadherence occurs when participants do not follow their assigned intervention or comparison condition as intended.
These problems may reduce the difference in actual exposure between groups. However, their implications depend on the causal effect being estimated.
In a properly randomized trial, an intention-to-treat analysis can preserve the causal interpretation of the effect of assignment despite nonadherence, assuming other necessary conditions hold. It does not necessarily identify the effect of actually receiving the intervention.
By contrast, comparing only participants who complied with their assignments may introduce selection bias if adherence is associated with prognosis or potential outcomes.
Researchers should distinguish contamination between study groups from failure to follow an assigned intervention.
Confounding Can Offer an Alternative Causal Explanation
Confounding occurs when the association between an exposure and an outcome is distorted by differences in other causes of the outcome.
Suppose researchers observe that students who use educational software perform better academically. Prior achievement may influence both software use and subsequent performance.
Unless the study addresses this relationship, the association may partly reflect prior achievement rather than a causal effect of software use.
Not every variable associated with the outcome is a confounder. Whether adjustment is appropriate depends on the variable's causal role, including whether it precedes the exposure or is affected by it.
Adjusting indiscriminately for variables can introduce bias, particularly when conditioning on mediators or colliders.
Testing Effects and Other Design-Specific Threats Also Matter
Repeated testing can influence later performance. Participants may remember questions, become familiar with the assessment format, or change their behavior because they have been measured.
Such testing effects are particularly relevant when the same instrument is administered repeatedly and there is no suitable comparison for practice effects.
Other concerns include deviations from intended interventions, outcome reporting decisions, and interactions among threats. For example, baseline selection differences may combine with maturation when groups are at different developmental stages.
These categories are diagnostic tools rather than an exhaustive taxonomy of every possible bias.
Which Threats Are Most Relevant to Different Research Designs?
| Research design |
Threats requiring particular attention |
Potential safeguards |
| Single-group pretest-posttest |
History, maturation, testing, regression to the mean, measurement changes |
Appropriate comparison groups, repeated observations, consistent measurement |
| Nonrandomized comparison-group study |
Selection differences, confounding, differential trends, attrition |
Careful group selection, baseline data, defensible causal adjustment strategies |
| Randomized controlled trial |
Allocation problems, attrition, deviations from assignment, contamination, outcome measurement bias |
Allocation concealment, appropriate randomization, follow-up procedures, prespecified analysis |
| Interrupted time-series study |
Concurrent events, changing trends, seasonality, measurement changes |
Multiple preintervention observations, suitable time-series models, comparison series where feasible |
| Longitudinal observational study |
Time-varying confounding, attrition, changes in exposure and measurement |
Repeated covariate measurement, appropriate longitudinal methods, missing-data assessment |
No design automatically eliminates every threat. A randomized study may offer strong protection against baseline confounding but still suffer serious bias from missing outcomes or flawed implementation. A nonrandomized study may support credible causal inference when its design, assumptions, and evidence adequately address the relevant alternative explanations.
Internal Validity Is Not Established by Statistical Significance
A small p-value indicates something about the compatibility of observed data with a specified statistical model and null hypothesis. It does not establish that a comparison is causally valid.
A biased estimate can be statistically precise. Increasing sample size may narrow confidence intervals around an estimate without removing the underlying bias.
Similarly, adding control variables does not automatically repair selection problems, measurement bias, or unmeasured confounding.
Internal validity therefore requires design-based reasoning as well as appropriate statistical analysis.
04 · A Practical Example
How Several Internal Validity Threats Can Affect One Educational Intervention
Hypothetical Example
Evaluating an AI-Supported Learning Program
A university introduces an AI-supported learning platform to one class of 40 students. Another class of 40 students continues using conventional learning materials. At the end of the semester, the intervention class achieves a higher mean examination score.
The researchers initially interpret the difference as evidence that the platform improves learning.
Closer examination identifies several plausible alternative explanations.
Selection: Were the groups comparable at baseline?
The intervention class had a higher average score on a prior assessment. Some of the final difference may reflect earlier academic preparation.
History: Did another event affect one group?
The intervention class also received additional examination preparation sessions. Their effect cannot be separated from the platform using the final scores alone.
Attrition: Who contributed outcome data?
Five lower-performing students in the intervention class did not complete the final examination, while nearly all comparison students did. The observed group means may therefore be affected by selective missing outcomes.
Measurement: Were outcomes assessed comparably?
The two classes completed different examination versions without evidence that the versions were equivalent in difficulty.
Interpretation: What conclusion remains defensible?
The observed difference establishes that the measured outcomes differed among students with available scores. It does not, by itself, establish how much of that difference was caused by the platform.
The researchers might improve a future evaluation by assigning participants randomly where feasible, using common outcome measures, maintaining comparable instructional conditions, and collecting outcomes for participants regardless of intervention engagement.
For the existing study, they should examine baseline differences, document co-interventions, assess missing outcomes, and evaluate whether the assessment results are comparable. Statistical adjustment or sensitivity analysis may help address some concerns, but cannot automatically reconstruct an unbiased comparison.
The appropriate conclusion depends on what the available evidence can support. A positive association may remain informative even when a strong causal claim is not justified.
