03 · What You Need to Know
Designing Sensitivity Analyses That Meaningfully Test Research Conclusions
What Is a Sensitivity Analysis?
A sensitivity analysis examines how a research conclusion changes when specified assumptions, analytical choices, or uncertain parameters are varied.
For example, researchers may estimate an intervention effect under different assumptions about missing outcomes or evaluate how an observational association would change under plausible levels of unmeasured confounding.
The purpose is to investigate the dependence of conclusions on assumptions that cannot be established with complete certainty.
Sensitivity analyses are particularly valuable when different defensible assumptions could produce substantively different interpretations.
However, conducting numerous alternative models does not automatically establish robustness. Each analysis should address an identifiable source of uncertainty and have a defensible scientific rationale.
How Does Sensitivity Analysis Differ From the Primary Analysis?
Primary analysis
Implements the principal analytical strategy selected to address the study's main research question under its specified assumptions.
Sensitivity analysis
Examines how the primary conclusion depends on alternative assumptions or analytically consequential decisions.
The distinction concerns analytical purpose rather than statistical complexity.
A sensitivity analysis may use a more complicated model than the primary analysis, but complexity alone does not make it more credible.
Researchers should establish which analysis is primary and explain what uncertainty each sensitivity analysis investigates.
How Does Sensitivity Analysis Differ From a Robustness Check?
The terms are sometimes used interchangeably, particularly outside regulated clinical research.
However, sensitivity analysis more explicitly concerns the consequences of varying assumptions, while robustness checks may refer more broadly to examining alternative specifications, measurements, or analytical approaches.
In the ICH E9(R1) framework, sensitivity analyses are specifically intended to assess robustness to assumptions underlying the main estimator for a particular estimand.
Supplementary analyses may instead investigate related questions or alternative estimands.
This distinction is important because changing the scientific question is not the same as testing whether an answer remains stable under different assumptions.
Which Assumptions Commonly Require Sensitivity Analysis?
| Source of uncertainty |
Possible sensitivity analysis |
Important qualification |
| Missing outcome data |
Evaluate plausible departures from the primary missingness assumptions. |
Alternative methods should correspond to meaningful missingness scenarios. |
| Unmeasured confounding |
Quantify how hypothetical confounding could affect an observational estimate. |
Such analyses do not prove that confounding is absent. |
| Measurement error |
Examine plausible misclassification or measurement error scenarios. |
Assumed error rates require justification. |
| Model specification |
Compare defensible functional forms or estimation methods. |
Alternatives should address a defined modeling uncertainty. |
| Influential observations |
Assess the influence of observations under justified diagnostic criteria. |
Exclusions should not be chosen because they improve significance. |
| Outcome definitions |
Examine justified alternative operational definitions. |
Different outcomes may represent different estimands or constructs. |
| Selection or attrition |
Evaluate assumptions about differential inclusion or loss to follow-up. |
Selection mechanisms may not be identifiable from observed data alone. |
Researchers should prioritize uncertainties that are both scientifically plausible and capable of materially affecting the intended conclusion.
Why Should Sensitivity Analyses Be Considered During Research Design?
Planning sensitivity analyses early can reveal information that the primary analysis alone would not require.
For example, evaluating the effects of missing follow-up data may require collecting baseline predictors of attrition and outcomes.
Similarly, assessing measurement error may benefit from a validation subsample or repeated measurements.
If these requirements are identified only after data collection, the necessary information may be unavailable.
Early planning also helps researchers anticipate whether the study's conclusions will depend heavily on assumptions that cannot be examined adequately.
In some cases, this may justify revising the study design rather than relying on statistical sensitivity analyses to compensate for preventable weaknesses.
Should Every Assumption Have a Sensitivity Analysis?
No. Some assumptions are relatively well supported, while others may have little influence on the primary conclusion.
Furthermore, certain assumptions cannot be meaningfully varied using the available data.
Researchers should prioritize assumptions according to their plausibility, importance, uncertainty, and potential consequences.
A useful question is whether a reasonable departure from the assumption could change the substantive interpretation of the result.
If so, a planned sensitivity analysis may be particularly valuable.
However, the number of analyses should not become a substitute for carefully identifying the vulnerabilities of the design.
How Should Sensitivity Analyses Be Selected?
Selection should begin with the assumptions underlying the primary analysis rather than a list of available statistical techniques.
Researchers should identify the uncertainty, describe a scientifically plausible alternative, and specify how the alternative will affect the analysis.
For example, a longitudinal study may assume that missing outcomes are missing at random conditional on observed information.
A sensitivity analysis could examine departures from that assumption using a prespecified range of shifts in imputed outcomes.
The range should reflect substantive knowledge or a transparent scenario-based rationale rather than values chosen after inspecting which assumptions preserve statistical significance.
How Should Missing-Data Sensitivity Analyses Be Planned?
Missing-data sensitivity analyses are particularly important when conclusions depend on assumptions about unobserved outcomes.
Methods such as multiple imputation or likelihood-based analysis often rely on assumptions that cannot be fully verified from the observed data.
For example, missing at random assumes that missingness can be explained by observed information, conditional on the model.
Researchers may examine missing-not-at-random scenarios using pattern-mixture models, selection models, or other suitable approaches.
However, these methods require assumptions about information that is not observed. Their results should therefore be interpreted as conditional on the scenarios examined.
Planning missing-data handling before data collection can help ensure that the study records relevant predictors of missingness.
Can Sensitivity Analysis Address Unmeasured Confounding?
Yes, in the sense that researchers can quantify how hypothetical unmeasured confounding might affect an estimated association or causal effect.
For example, an observational study may estimate an association between educational technology use and academic performance after adjusting for measured baseline characteristics.
A sensitivity analysis can examine how strongly an unmeasured factor would need to relate to both exposure and outcome to explain away the estimated association under specified assumptions.
Methods such as quantitative bias analysis and the E-value provide approaches for examining particular confounding scenarios.
However, they do not establish that unmeasured confounding is absent or that the observed association is causal.
Researchers should select methods appropriate to the effect measure and causal assumptions involved.
Should Alternative Statistical Models Be Included?
Alternative models may be useful when the primary result depends on uncertain functional forms, covariance assumptions, or estimation procedures.
For example, researchers may examine whether an estimated intervention effect is sensitive to a defensible alternative approach for handling classroom clustering.
However, comparing models that estimate different scientific quantities may not constitute a sensitivity analysis of the same estimand.
Researchers should explain whether each alternative tests a modeling assumption or answers a different question.
Model selection should not be driven solely by which specification produces the smallest p-value.
Can Sensitivity Analyses Address Measurement Error?
Yes. Measurement error can distort estimates, particularly when its magnitude or direction differs across participants or groups.
For example, self-reported time spent using an educational platform may differ from actual usage recorded by the system.
Researchers may examine plausible error distributions or use validation information to evaluate how estimates change.
However, the credibility of the analysis depends on the assumptions about measurement error.
Where possible, a validation subsample or independent measurement may strengthen the analysis.
Planning these procedures before data collection can be more informative than attempting to infer measurement error entirely from the final dataset.
Can Sensitivity Analyses Address Outliers?
Potentially, but researchers should distinguish genuine observations from data errors and influential values.
An extreme observation is not necessarily invalid.
For example, a student may legitimately have an unusually high or low assessment score.
Researchers can examine how justified alternative treatments of influential observations affect results, but exclusions should follow defensible criteria.
Removing observations solely because they weaken a preferred conclusion is not a legitimate robustness assessment.
The primary analysis should remain clearly distinguished from alternative analyses involving modified datasets.
What Is a Tipping-Point Analysis?
A tipping-point analysis examines how far an uncertain assumption must change before a specified conclusion changes.
For example, researchers may investigate how unfavorable missing outcomes would need to be before an estimated intervention effect loses statistical significance or crosses a substantively important threshold.
However, statistical significance is not the only possible criterion for a tipping point.
A more substantively useful analysis may examine when the estimated effect falls below a threshold of practical importance.
The plausibility of the tipping-point assumptions must still be evaluated.
A conclusion that changes only under extreme assumptions may be less sensitive than one that changes under modest and credible departures from the primary assumptions.
Should Sensitivity Analyses Be Powered Separately?
There is no universal requirement to conduct a separate power calculation for every sensitivity analysis.
However, anticipated robustness assessments can influence sample-size and data collection planning.
For example, a validation subsample intended to estimate measurement error may require its own precision considerations.
Similarly, analyses relying on rare outcomes or small subgroups may be too imprecise to provide useful evidence.
Researchers should evaluate whether the available data can meaningfully distinguish the scenarios under investigation.
Sensitivity analysis does not create information that the study failed to collect.
Should Sensitivity Analyses Be Prespecified?
Important sensitivity analyses should generally be prespecified when they support the credibility of confirmatory conclusions.
The statistical analysis plan should identify the assumption being varied, the alternative scenarios, the analytical method, and the intended interpretation.
However, unforeseen problems may justify additional analyses after data collection.
These should be reported transparently, with attention to when they were developed and what information was available.
Prespecification reduces opportunities for selective presentation but does not guarantee that the selected scenarios are scientifically appropriate.
How Should Sensitivity Analyses Be Interpreted When Results Agree?
Agreement across analyses may increase confidence that the conclusion is not highly dependent on the particular assumptions examined.
However, several analyses may share the same underlying weakness.
For example, multiple regression specifications may all remain vulnerable to the same unmeasured confounder.
Therefore, consistency across models does not establish that the findings are unbiased or causally valid.
Researchers should describe the scope of robustness demonstrated rather than claim that all possible sources of uncertainty have been eliminated.
What If Sensitivity Analyses Produce Different Conclusions?
Disagreement can be scientifically informative because it identifies assumptions that materially influence the result.
For example, an intervention effect may remain favorable under missing-at-random assumptions but become uncertain under plausible missing-not-at-random scenarios.
Researchers should explain which assumptions produce the different conclusions and evaluate their plausibility.
They should not automatically select the most favorable or most conservative estimate as the definitive answer.
The primary analysis remains important, but the sensitivity results may require a more qualified interpretation of its findings.
Can Sensitivity Analyses Replace a Strong Research Design?
No. Sensitivity analyses help evaluate the consequences of assumptions but cannot reliably compensate for every design weakness.
For example, if a study lacks essential confounder information, a sensitivity analysis may quantify hypothetical bias but cannot recover the missing variables.
Likewise, severe selection bias or inappropriate measurement may prevent defensible inference even when several alternative models produce similar estimates.
Researchers should use sensitivity planning to identify weaknesses that can be addressed through better sampling, measurement, follow-up, or allocation procedures.
The objective is to improve the design where possible and assess residual uncertainty where it cannot be eliminated.
How Should Sensitivity Analyses Be Documented?
The analysis plan should identify the primary analysis, the uncertainty addressed by each sensitivity analysis, the alternative assumptions, and the corresponding estimation procedures.
It should also describe relevant parameters, plausible ranges, and criteria for interpreting changes in conclusions.
Where additional analyses are developed later, their rationale and timing should be recorded.
Reports should present important sensitivity results even when they weaken the apparent strength of the primary finding.
The appropriate detail of the statistical analysis plan depends on the inferential role of these analyses.
Watch Out
Do not conduct numerous alternative models and report only those that preserve the preferred conclusion. A defensible sensitivity analysis should address a stated uncertainty, use scientifically meaningful alternatives, and disclose results that materially change the interpretation.
