03 · What You Need to Know
How to Determine Whether Difference-in-Differences Fits a Research Question
What difference-in-differences actually estimates
Difference-in-differences, commonly abbreviated DiD, compares the change in an outcome for a treated group with the corresponding change for a comparison group.
Its basic logic is that some outcome changes would occur even without the intervention. By subtracting the comparison group's change, researchers attempt to remove the portion of the treated group's change attributable to shared background influences.
For example, student retention may improve across universities because of economic conditions, changes in student support, or broader educational policies. A suitable comparison group can provide information about those changes.
However, the method does not automatically isolate the intervention effect. Its causal interpretation depends on whether the comparison group's observed change represents the treated group's counterfactual change.
The basic two-group, two-period calculation
The conventional difference-in-differences estimator subtracts the comparison group's before-and-after change from the treated group's before-and-after change.
The six-percentage-point estimate represents the additional observed improvement in the treated group relative to the comparison group.
It can be interpreted as a causal effect under the relevant identification assumptions. Without those assumptions, it remains a descriptive contrast between changes.
When the research question concerns an intervention introduced at a known time
Difference-in-differences is particularly useful for evaluating interventions, policies, or institutional changes introduced to some units but not others.
Examples include universities adopting new student support programs, schools implementing curriculum reforms, hospitals introducing revised clinical protocols, or regions receiving government assistance.
The intervention should be sufficiently well defined that researchers can identify who was treated, when treatment began, and what comparison condition existed.
Uncertain implementation dates or highly variable exposure can complicate interpretation.
Researchers should distinguish formal program announcement, operational implementation, and meaningful exposure when those events occur at different times.
Why a comparison group is necessary
Conventional difference-in-differences requires observations from treated and comparison units.
The comparison group supplies information about outcome changes that might have occurred without the intervention.
However, untreated status alone does not make a group suitable.
For example, universities that do not adopt an AI advising system may differ from adopters in financial resources, student characteristics, institutional leadership, or existing retention trends.
Researchers should investigate whether these differences threaten the intended counterfactual comparison.
The criteria for a credible quasi-experimental comparison group depend on the identification strategy rather than baseline similarity alone.
The parallel trends assumption is central
The parallel trends assumption concerns how the treated and comparison groups would have changed over time if the intervention had not occurred.
In the conventional two-group, two-period setting, it requires equality of the groups' average untreated outcome changes over the relevant period.
This does not require identical baseline outcome levels.
For example, one university may consistently have lower retention than another while both experience similar changes over time. Such differences in levels may be compatible with parallel trends.
Conversely, two universities may have identical retention immediately before treatment but substantially different underlying trajectories.
The assumption concerns counterfactual changes, not merely observed baseline equality.
The parallel trends assumption in difference-in-differences therefore requires careful substantive and empirical assessment.
Why parallel trends cannot be proven from the observed data
The treated group's untreated potential outcomes after intervention are not observed.
Consequently, researchers cannot directly verify whether the treated and comparison groups would have followed parallel trends without treatment.
Several pre-intervention observations can help assess whether historical trajectories were similar and whether important differences were already emerging.
However, similar pre-intervention trajectories do not guarantee that trends would have remained parallel after implementation.
A nonsignificant pre-trend test is also not proof of equivalence. Such tests may have limited power to detect meaningful differences.
Researchers should combine graphical examination, substantive knowledge, institutional histories, and appropriate sensitivity analyses when evaluating the assumption.
When different baseline levels are acceptable
Difference-in-differences can accommodate stable differences in outcome levels between groups.
Suppose University A has retention rates of 70%, 74%, and 78% across three pre-intervention years, while University B has rates of 80%, 84%, and 88%.
Although University B consistently has higher retention, both universities improve by four percentage points annually.
These observed patterns may support a parallel-trends argument, although they do not establish future counterfactual trends.
The distinction is important because selecting comparison groups solely according to the latest baseline mean may produce a weaker comparison than examining their trajectories.
When group composition must remain comparable
Difference-in-differences can use panel data following the same units or repeated cross-sectional data observing different individuals over time.
In either case, researchers must consider whether changes in the observed population influence the outcome comparison.
For example, a university may change its admissions policy at the same time it introduces an AI advising system. The characteristics of incoming students may then differ from those observed before implementation.
Observed changes in retention could partly reflect changes in student composition rather than the advising intervention.
Researchers should define eligibility and sampling consistently, measure relevant pretreatment characteristics, and consider whether compositional changes threaten identification.
Adjustment for variables affected by treatment requires additional care because it may change the estimand or introduce bias.
Why concurrent interventions can invalidate the comparison
Suppose the treated university introduces an AI advising system and expands financial assistance during the same academic year.
If the comparison university does not introduce similar assistance, its outcome change may not represent the treated university's counterfactual trend.
The estimated difference-in-differences effect could reflect both interventions.
Researchers should document other policies, institutional changes, and external events occurring during the evaluation period.
If two interventions are inseparable in the observed data, the estimand may concern their combined effect rather than the effect of either component individually.
Anticipation can compromise pretreatment observations
Participants and institutions may change their behavior before the official intervention date if they know treatment is forthcoming.
For example, teachers may begin preparing instructional materials before a digital platform becomes operational. Administrators may reorganize advising procedures after a new system is announced.
If these changes affect outcomes, periods classified as untreated may already contain intervention-related effects.
Researchers should investigate announcement dates, preparatory activities, and implementation lags.
Where justified, anticipation periods can be explicitly modeled or excluded from particular comparisons according to a prespecified strategy.
Why treatment spillover matters
Difference-in-differences commonly assumes that the comparison group's outcomes are not affected by the treated group's intervention in ways that undermine the intended estimand.
However, interventions can produce spillover.
For example, teachers at untreated schools may adopt instructional strategies shared by colleagues at treated schools.
Alternatively, a regional policy may affect neighboring institutions through student transfers, resource competition, or information sharing.
Such interference can change comparison outcomes and complicate the causal interpretation.
Researchers should assess whether the chosen comparison group remains meaningfully untreated and whether indirect effects are part of the research question.
When two periods are enough and when more are useful
A basic difference-in-differences estimate can be calculated using one pretreatment and one post-treatment period for each group.
However, additional pretreatment periods are often valuable because they allow researchers to examine historical trajectories and identify potential deviations before intervention.
Multiple post-treatment periods can reveal whether effects develop, persist, or diminish.
More observations do not automatically establish parallel trends or eliminate confounding, but they can improve the assessment of temporal patterns.
The number and spacing of periods should reflect the outcome's natural timescale, intervention implementation, and analytical objectives.
When staggered adoption requires different estimators
In many policy evaluations, institutions adopt an intervention at different times rather than simultaneously.
These settings are commonly called staggered adoption designs.
Conventional two-way fixed-effects regressions may produce difficult-to-interpret weighted averages when treatment effects vary across adoption cohorts or exposure durations.
Some comparisons may implicitly use already-treated units as controls for newly treated units.
Callaway and Sant'Anna developed methods for estimating group-time average treatment effects, while Sun and Abraham examined event-study estimation under heterogeneous treatment effects.
These approaches can provide more transparent estimates when their identification assumptions are satisfied.
Researchers should specify whether comparison units are never treated or not yet treated, how cohort-specific effects are estimated, and how effects are aggregated.
The fact that an intervention was introduced in stages does not justify applying a conventional two-way fixed-effects model without examining these issues.
How to interpret the estimand
Difference-in-differences commonly targets an average treatment effect for the treated population under the relevant assumptions.
However, the precise estimand depends on the study design, treatment timing, comparison group, and aggregation procedure.
For example, an evaluation of AI advising adoption may estimate the average effect among universities that implemented the program, rather than the effect that would occur at every university.
When effects differ across institutions, exposure durations, or student populations, the target population matters.
Researchers should state the estimand before estimation and avoid interpreting a cohort-specific or local effect as universally applicable.
Why statistical significance does not establish validity
A statistically significant difference-in-differences estimate may still be biased if the parallel trends assumption is implausible or concurrent interventions affect treated and comparison groups differently.
Large datasets can produce precise estimates of invalid comparisons.
Conversely, a credible design may produce an imprecise estimate because the number of institutions or the available identifying variation is limited.
Researchers should distinguish statistical precision from causal identification.
When treatment is assigned at the school, university, or regional level, standard errors must also account appropriately for the dependence structure. Having many individual observations does not automatically provide many independent treatment-assignment units.
What diagnostics and sensitivity analyses are useful?
Researchers should examine pretreatment outcome trajectories and investigate whether major concurrent changes differ between groups.
Event-study analyses may help describe effects relative to treatment timing, but their interpretation depends on the estimator and assumptions.
Placebo intervention dates or negative control outcomes may reveal particular threats when substantively justified.
Sensitivity analyses can assess how conclusions change under plausible violations of parallel trends or alternative comparison definitions.
However, passing diagnostic checks does not prove that all assumptions hold.
The strongest evaluation combines a defensible assignment narrative, appropriate estimation, and transparent assessment of remaining uncertainty.
When difference-in-differences is not appropriate
Difference-in-differences may be inappropriate when no credible comparison group exists, treatment timing is poorly defined, or treated and comparison units would plausibly have followed substantially different counterfactual trends.
It may also be unsuitable when major concurrent interventions cannot be separated, measurement procedures change inconsistently, or anticipation and spillover undermine the comparison.
In such circumstances, researchers should consider whether regression discontinuity, interrupted time series, or another design better matches the assignment mechanism.
If no defensible identification strategy is available, descriptive or associational conclusions may be more appropriate.
Watch Out
Difference-in-differences does not become causal merely because the calculation subtracts two changes. Its validity depends on whether the comparison group's outcome change represents the treated group's counterfactual change under the specified assumptions.
