How Do You Identify the Most Serious Threat to Internal Validity in Your Research Design?

Which Internal Validity Threat Should You Be Most Concerned About?

You have designed an intervention study and identified several possible threats to internal validity. The groups may differ at baseline, some participants may discontinue the intervention, and a few outcome measurements may be missing. External events could also influence the results.

Should you treat all these threats as equally serious? Should you discuss every possible source of bias in your methodology, or concentrate on the problems most likely to change your conclusions?

A list of potential threats is a useful starting point, but it does not establish which threats are consequential. The central task is to determine which features of the actual research design could materially distort the causal comparison supporting the study's main claim.

How Do You Determine the Most Serious Internal Validity Threat?

A Design-Based Approach to Identifying Consequential Validity Threats

Begin With the Causal Claim, Not a List of Threats

Internal validity concerns whether a study provides credible evidence for its intended causal conclusion. Before assessing threats, researchers must establish precisely what that conclusion is.

Consider a study evaluating whether an adaptive learning platform improves students' examination performance compared with conventional instruction.

The relevant questions include who receives the intervention, what the comparison condition involves, when outcomes are measured, and which population the estimated effect concerns.

The causal estimand should be specified as clearly as possible. An estimand identifies the effect being targeted, including the population, treatment conditions, outcome, and relevant treatment strategies.

For example, the effect of being assigned access to a platform differs from the effect of completing every platform activity. A design may identify one effect credibly while providing limited evidence about the other.

Without a clear estimand, researchers may label an implementation issue a serious threat even when it does not undermine the causal quantity actually being estimated.

Identify the Comparison That Makes the Causal Claim Possible

Every causal comparison requires some basis for estimating what would have happened under an alternative condition.

In a randomized experiment, this basis may be provided by participants assigned to another condition. In a nonrandomized study, it may involve a carefully selected comparison group, a discontinuity around an assignment threshold, or a counterfactual trend estimated from repeated observations.

Ask what makes the comparison credible.

If the argument depends on random assignment, examine whether randomization was implemented correctly and whether subsequent analysis preserved the relevant comparison.

If the argument depends on parallel trends, examine why the untreated outcome trajectories should be comparable.

If it depends on measured confounding control, identify which common causes of exposure and outcome were measured and whether the adjustment strategy is defensible.

The most consequential threat is often the one that challenges the assumption on which the causal comparison most strongly depends.

Which Threats Deserve Attention in Different Research Designs?

Different designs create different opportunities for bias. A threat that dominates one design may be relatively well controlled in another.

This table identifies areas requiring investigation rather than assigning automatic judgments. The actual risk depends on how the study was conducted.

Researchers should first understand the main mechanisms that can undermine causal comparisons, then determine which mechanisms are relevant to their own design.

Distinguish a Possible Threat From Evidence of Actual Bias

A possible threat is a mechanism through which bias could occur. Evidence of bias requires additional information about whether that mechanism operated and how it affected the estimate.

Suppose a study loses 15% of participants during follow-up. This raises a question about missing outcome data, but the percentage alone does not establish that the estimated effect is biased.

The seriousness depends on whether missingness is associated with treatment assignment, prognostic characteristics, or the outcomes that would otherwise have been observed.

Similarly, a baseline difference between randomized groups may arise by chance without indicating defective randomization.

Researchers should avoid treating every observable irregularity as proof of bias. They should also avoid treating an absence of statistically significant differences as proof that bias is absent.

A defensible assessment identifies the mechanism, the evidence supporting concern, and the remaining uncertainty.

How Do You Evaluate the Plausibility of a Bias Mechanism?

Begin by explaining how the proposed threat could influence the estimated effect.

For confounding, identify a variable or process that influences both intervention assignment and the outcome.

For outcome measurement bias, explain how the assessment procedure could produce systematically different recorded values across groups.

For attrition, consider how the probability of observing an outcome might depend on the outcome itself or on related prognostic factors.

For history effects, identify external events occurring during the observation period that could influence the outcome independently of the intervention.

A threat is more persuasive when supported by a plausible causal pathway and relevant evidence.

For example, an institution-wide examination preparation program introduced during a teaching intervention may plausibly influence academic outcomes. An unrelated administrative change with no credible connection to learning performance may be less consequential.

The assessment should be specific to the study rather than based on generic statements that external factors might exist.

Can Causal Diagrams Help Identify Important Threats?

Directed acyclic graphs, commonly called DAGs, can help researchers represent assumptions about causal relationships among variables.

A DAG may identify common causes of treatment and outcome, mediating pathways, and selection mechanisms that could introduce bias if researchers condition on particular variables.

For example, prior academic achievement may influence both voluntary platform use and subsequent examination performance. Representing these relationships helps explain why prior achievement may be a confounder.

A DAG may also reveal why adjusting for a variable affected by treatment could introduce bias or change the estimand.

However, a causal diagram does not establish that its assumptions are correct. Its value depends on substantive knowledge and the accuracy of the relationships represented.

Researchers should use diagrams to make causal assumptions explicit, not to replace empirical investigation.

How Do You Evaluate the Potential Magnitude of a Threat?

The magnitude of bias concerns how far the estimated effect may be from the causal quantity of interest.

Researchers rarely know the exact magnitude of an unobserved bias. Nevertheless, they can investigate whether plausible bias scenarios would materially change the conclusion.

For example, if the estimated intervention advantage is two examination points, a plausible measurement difference of several points could be consequential.

If the observed effect is much larger, the same measurement discrepancy may be less influential, although other sources of bias could remain.

Researchers should consider the outcome scale, estimated effect, uncertainty, and substantive importance of possible changes.

Bias direction also matters. Some mechanisms may exaggerate effects, others may diminish them, and several may operate in opposing directions.

Do not assume that multiple biases cancel each other. Their combined consequences generally cannot be inferred without additional evidence and assumptions.

Why Should You Examine Existing Design Safeguards?

The seriousness of a threat depends partly on what the research design has already done to address it.

For example, proper randomization provides protection against systematic baseline confounding in expectation. It does not guarantee protection against missing outcomes or differential outcome measurement after assignment.

Similarly, a concurrent comparison group may help address common history effects, but differential exposure to external events may remain.

Researchers should therefore evaluate residual risk after considering the safeguards actually implemented.

It is insufficient to state that a study is randomized, longitudinal, or quasi-experimental without examining whether the relevant design assumptions were satisfied.

How Can Selection Differences Become the Dominant Threat?

Selection differences may be particularly serious when intervention assignment is strongly related to characteristics that also determine outcomes.

Suppose academically advanced students voluntarily enroll in an intensive learning program, while students with weaker preparation remain in conventional instruction.

If prior achievement is not adequately addressed, the observed outcome difference may substantially overstate the program's benefit.

Even extensive statistical adjustment may leave residual confounding when important prognostic characteristics are unmeasured.

In such a study, preexisting differences between intervention and comparison groups may deserve greater attention than relatively minor procedural inconsistencies.

When Might History or Maturation Be the Main Concern?

In a single-group before-and-after study, the absence of a credible counterfactual may make time-related alternative explanations particularly consequential.

Participants may improve because of ordinary development, accumulated experience, or recovery. External events may also influence the outcome during follow-up.

Researchers should distinguish history effects arising from concurrent events from maturation-related changes occurring over time.

Neither mechanism can ordinarily be eliminated merely by demonstrating a statistically significant pretest-posttest difference.

If participants were selected because of extreme baseline scores, regression to the mean may provide another substantial alternative explanation.

When Can Missing Outcomes Become More Serious Than Baseline Confounding?

Consider a randomized trial with a well-implemented allocation process but substantial missing outcome data.

Baseline confounding may be adequately controlled through randomization in expectation. However, selective outcome observation can undermine the comparison among participants whose outcomes are available.

If students who struggle with the intervention disproportionately miss the final assessment, the observed intervention mean may be misleading.

In this setting, missing outcome data and their underlying mechanisms may deserve greater attention than chance baseline imbalances.

The most important threat is determined by the actual study process, not by a fixed hierarchy of methodological concerns.

How Do Formal Risk-of-Bias Frameworks Help?

Formal frameworks provide structured ways to evaluate bias mechanisms and document judgments.

The Cochrane RoB 2 tool assesses risk of bias in randomized trials through domains involving randomization, deviations from intended interventions, missing outcome data, outcome measurement, and selection of reported results.

For nonrandomized studies of interventions, ROBINS-I addresses domains including confounding, participant selection, intervention classification, deviations from intended interventions, missing data, outcome measurement, and selection of reported results.

These frameworks are designed for particular study types and effect estimates. They should not be applied mechanically to every research question.

Importantly, formal risk-of-bias assessments generally do not treat all domains as interchangeable or calculate validity by simply averaging numerical scores.

A serious problem in one consequential domain may substantially limit confidence in the effect estimate even when other domains are relatively well addressed.

Researchers should consult the current official guidance for the framework appropriate to their design.

Should You Create a Numerical Ranking of Internal Validity Threats?

A numerical priority score may be useful for internal project planning, but it should not be mistaken for a validated measure of bias.

Assigning arbitrary values for likelihood, severity, and detectability can create an appearance of precision unsupported by the evidence.

For example, multiplying subjective ratings does not establish how many examination points a particular threat might add to or subtract from an estimated effect.

A qualitative evidence-based assessment is often more defensible.

Researchers can describe each threat's causal mechanism, supporting evidence, existing safeguards, plausible effect on the estimate, and unresolved assumptions.

When quantitative bias analysis is feasible, it should use explicit parameters and scientifically justified assumptions rather than arbitrary scores.

How Should Multiple Threats Be Considered Together?

Internal validity threats may interact.

For example, students who select into an intervention may differ at baseline, and those who struggle may subsequently leave the study. The final comparison may therefore involve both confounding and selective missing outcomes.

Measurement procedures may also change during the intervention period, adding another source of uncertainty.

Researchers should not evaluate each threat as though the others were absent when their mechanisms are related.

A useful approach is to reconstruct the sequence of events from recruitment and assignment through intervention delivery, measurement, follow-up, and analysis.

This makes it easier to identify where bias could enter and whether one problem influences another.

What Evidence Should Support Your Final Judgment?

A defensible judgment should be traceable to the study protocol, implementation records, participant flow, measurement documentation, and analytical decisions.

Relevant evidence may include allocation procedures, baseline characteristics, adherence records, missing-data patterns, calibration logs, assessment rubrics, and prespecified analysis plans.

Researchers should also consider substantive knowledge about the outcome and the population.

Where uncertainty cannot be resolved, the appropriate response is to describe it rather than manufacture confidence.

The objective is not to prove that a study has no weaknesses. It is to determine whether the evidence supports the causal conclusion and which limitations most materially affect that judgment.

Identifying the Dominant Validity Threat in an Educational Intervention

The researchers identify several possible threats and examine their relevance.

In this hypothetical study, missing outcome data emerge as the leading concern because the evidence identifies a plausible mechanism capable of materially distorting the assignment comparison.

This does not prove that missingness caused bias or that every other threat is negligible.

The judgment is conditional on the available evidence and should be revised if additional information reveals another substantial problem.

Common Mistakes When Prioritizing Internal Validity Threats

How to Conduct a Focused Internal Validity Assessment

Start with a concise statement of the causal effect being estimated. Then reconstruct the study process and identify the assumptions required for the main comparison.

For each plausible threat, document the mechanism, available evidence, safeguards, and remaining uncertainty.

Prioritize problems that could materially change the estimated effect or invalidate the interpretation, particularly when the design provides limited protection against them.

For planned research, use the assessment to determine which safeguards should be incorporated before recruitment and data collection.

For completed research, distinguish weaknesses that can be investigated analytically from those that cannot be resolved with the available information.

The most useful assessment explains why a particular threat matters, what evidence supports concern, and how it affects the conclusion.

Checks for Identifying the Most Consequential Validity Threat

Frequently Asked Questions About Prioritizing Internal Validity Threats

The Most Serious Threat Is the One That Most Undermines Your Causal Comparison

How to Cite This Guide

This guide is intended to be read, shared, and used in research, teaching, and academic work. If you draw on its ideas, explanations, or other content, please acknowledge the source by citing the guide. Doing so gives appropriate credit and helps your readers locate the original resource.