03 · What You Need to Know
Understanding Cross-Design Replication and Causal Evidence Triangulation
What Is Replication Across Different Research Designs?
Cross-design replication involves investigating a related research proposition using studies that differ in their methodological approaches.
For example, researchers may examine an educational intervention through a randomized experiment, a natural experiment, and an observational longitudinal study.
These designs may provide different forms of evidence about the same proposed causal relationship.
However, the term replication requires care. Repeating a study with the same procedures differs from examining a proposition using different designs.
The latter is sometimes discussed in relation to triangulation or the integration of evidence from approaches with different assumptions.
Munafò and Davey Smith (2018) discuss robust research through triangulation, emphasizing the importance of approaches with differing and preferably unrelated sources of bias.
How Does Cross-Design Replication Differ From Repeating the Same Study?
Direct replication attempts to reproduce an investigation's central procedures and conditions closely enough to assess whether its findings recur.
Conceptual replication investigates a related theoretical proposition using different operationalizations, populations, or procedures.
Cross-design replication focuses particularly on whether different methodological approaches provide compatible evidence about a common proposition.
These categories can overlap, and terminology varies across disciplines.
For example, repeating the same randomized trial at another university may examine reproducibility across contexts. Comparing randomized and observational estimates may instead examine convergence across identification strategies.
Both can contribute to cumulative knowledge, but they answer somewhat different questions.
Why Can Different Designs Provide Stronger Combined Evidence?
Different research designs may be vulnerable to different sources of bias.
A randomized trial can address systematic baseline treatment selection in expectation but may face nonadherence, limited generalizability, or intervention implementation problems.
An observational study may examine a broader population under routine conditions but depend heavily on assumptions about confounding.
A regression discontinuity design may exploit an assignment threshold while primarily identifying a local effect near that threshold.
When findings from these designs are compatible, and their important biases are sufficiently different, the evidence may become harder to explain through one shared methodological artifact.
However, the strength of convergence depends on the actual independence of the biases and the comparability of the causal questions.
What Does Triangulation Mean in Causal Research?
Triangulation refers broadly to using multiple approaches to investigate a phenomenon or proposition.
In causal epidemiology and related methodological discussions, triangulation has a more specific emphasis on combining approaches with different and preferably unrelated sources of potential bias.
For example, researchers might examine a proposed causal effect through randomized assignment, an observational longitudinal design, and a credible natural experiment.
The value lies not simply in using several methods, but in understanding how each method could be wrong.
Lawlor, Tilling, and Davey Smith (2016) describe triangulation as a strategy for strengthening causal inference through evidence from approaches with different key sources of bias.
Triangulation does not automatically produce a single pooled estimate. Its primary contribution may be a more credible assessment of the causal explanation.
What Types of Evidence Might Be Combined?
| Research design |
Potential contribution |
Important limitation |
| Randomized experiment |
Supports causal inference about assigned intervention conditions. |
May face nonadherence, contamination, and limited external validity. |
| Prospective observational study |
Examines temporal associations under routine exposure conditions. |
May retain unmeasured confounding and selection bias. |
| Difference-in-differences |
Uses changes across groups and periods to investigate intervention effects. |
Depends on defensible counterfactual trend assumptions. |
| Regression discontinuity |
May identify a causal effect near an assignment threshold. |
Requires continuity and other design assumptions; effects may be local. |
| Interrupted time series |
Examines changes in outcome levels or trends around an intervention. |
May be affected by concurrent events and time-series model assumptions. |
| Qualitative process evaluation |
Investigates implementation, context, experiences, and plausible mechanisms. |
Does not independently identify an average causal effect merely by describing processes. |
These approaches need not all be included in one research program. The appropriate combination depends on the causal question, feasibility, and the specific uncertainties requiring investigation.
Does Agreement Between Studies Automatically Establish Causation?
No. Several studies may reproduce the same misleading association.
For example, multiple observational studies of AI feedback may all fail to measure academic motivation adequately.
If motivation influences both AI use and writing performance, each study may estimate a positive association partly attributable to the same confounder.
Using different regression specifications or collecting data at different universities does not necessarily eliminate that shared bias.
Agreement is more informative when the designs genuinely differ in how the principal alternative explanations could affect their results.
Researchers should therefore assess the structure of the evidence rather than count positive studies.
Why Does Independence of Bias Matter?
Suppose two studies rely on different identification strategies.
The first uses randomized assignment, while the second exploits a policy threshold that determines access to an intervention.
If both estimate compatible effects, and their main assumptions are credible, the agreement may be difficult to explain through a single confounding process.
However, both studies might use the same poorly validated writing assessment.
If the assessment systematically favors a particular instructional approach, their results may still share measurement bias.
Cross-design triangulation therefore requires examining both independent and shared vulnerabilities.
Methodological diversity is useful only when it changes the relevant routes through which bias could arise.
Must Different Studies Estimate Exactly the Same Effect?
Not necessarily, but differences in estimands must be recognized.
A randomized trial may estimate the effect of offering AI-supported feedback to eligible students.
An observational study may estimate the effect of sustained voluntary use among students with particular characteristics.
A regression discontinuity design may estimate an effect near an eligibility threshold.
These are related but not identical causal quantities.
Compatible findings may support a broader causal proposition, but their numerical estimates should not automatically be treated as interchangeable.
Researchers should compare the intervention definitions, populations, outcomes, and time horizons before interpreting convergence.
What If Different Designs Produce Different Effect Sizes?
Different effect estimates do not necessarily indicate that one study is wrong.
Effects may vary across populations, intervention implementations, exposure intensities, and outcome measurement periods.
For example, a randomized trial evaluating access to AI feedback may estimate a smaller effect than an observational study comparing highly engaged users with nonusers.
The difference may reflect confounding, adherence, treatment intensity, or genuinely different estimands.
Researchers should investigate these possibilities rather than assume that all estimates should be numerically identical.
Compatibility should be assessed with reference to uncertainty and substantive differences between studies.
How Should Conflicting Results Be Interpreted?
Disagreement across designs can be scientifically informative.
For example, observational studies may report large positive associations while randomized trials estimate little average effect of assignment.
This pattern may suggest observational confounding, differences in adherence, intervention definitions, or populations.
Alternatively, a trial may have limited precision or evaluate an intervention strategy different from the observational exposure.
Researchers should investigate why the results differ rather than automatically privilege the estimate with the smallest p-value.
Contradictory evidence can help identify assumptions requiring further scrutiny.
Does Replication Across Populations Strengthen Causal Interpretation?
Potentially. Findings observed across different populations may support the proposition that an effect is not restricted to one narrowly defined setting.
For example, an AI feedback intervention may be evaluated among students from different academic programs or institutions.
However, populations may differ in characteristics that modify the effect.
A positive effect in one population and a negligible effect in another may reflect genuine heterogeneity rather than failed replication.
Researchers should distinguish evidence about causal identification within a study from evidence about transportability across populations.
Replication across settings can strengthen generalizability while leaving some shared identification problems unresolved.
What Is the Difference Between Replication and Generalizability?
Replication concerns whether findings or propositions recur under specified conditions.
Generalizability concerns whether findings apply to populations or settings beyond those directly studied.
A randomized trial may provide credible evidence of an effect in one university without establishing that the effect will occur in every institution.
Repeating the trial across different settings may help assess contextual variation.
However, replication does not automatically establish representativeness, and a representative sample does not automatically establish causal validity.
Researchers should evaluate the two questions separately.
Can Qualitative Evidence Strengthen a Causal Interpretation?
Qualitative research may provide important evidence about implementation, participant experiences, contextual conditions, and proposed causal mechanisms.
For example, interviews may reveal whether students actually used AI feedback to revise their writing or merely accessed the platform.
Such findings may help interpret why an intervention effect was observed or why implementation differed across settings.
However, qualitative evidence describing a plausible mechanism does not independently identify an average causal effect.
Its contribution should be understood in relation to the causal question and the other evidence available.
Meaningful integration may strengthen interpretation without treating all forms of evidence as estimating the same quantity.
Does Mixed-Methods Research Automatically Provide Causal Triangulation?
No. Combining quantitative and qualitative components does not automatically create independent causal evidence.
For example, a survey showing that AI users report higher confidence and interviews describing positive experiences may both reflect self-selection or favorable perceptions.
The two components may complement one another, but they do not necessarily provide independent evidence that AI use caused improved learning.
Causal triangulation requires evaluating what each method establishes and whether its limitations differ meaningfully from those of the other methods.
Researchers should distinguish mixed-methods integration from the more specific objective of strengthening causal identification through complementary designs.
Can Replication of the Same Bias Create False Confidence?
Yes. Repeated findings can appear persuasive when studies share a common methodological weakness.
For example, several studies may measure educational engagement using platform login counts, even though logins poorly represent meaningful participation.
Consistent associations between logins and achievement may reflect a shared measurement limitation or confounding by general academic engagement.
Replicating the association does not automatically validate the underlying construct or causal explanation.
Researchers should examine whether the evidence sources are genuinely independent in the ways that matter for the proposed causal claim.
How Does Publication Bias Affect Cross-Design Evidence?
Publication and selective reporting can distort the apparent consistency of research findings.
Studies reporting statistically significant or favorable results may be more likely to appear in the accessible literature than studies reporting null or conflicting results.
Researchers may also selectively report outcomes, subgroups, or analytical specifications.
Consequently, a literature containing many positive studies may not represent the full body of conducted research.
Preregistration, registered reports, transparent protocols, and access to unpublished or nonsignificant findings can help evaluate the completeness of the evidence base.
Cross-design replication should consider reporting processes as well as methodological differences.
What Is the Role of Systematic Reviews and Meta-Analysis?
Systematic reviews can identify, appraise, and synthesize relevant studies across designs.
Meta-analysis may estimate a combined effect when studies are sufficiently compatible and statistical synthesis is justified.
However, pooling estimates does not automatically resolve confounding, selection bias, or incompatible estimands.
A precise pooled estimate can still be biased when the contributing studies share systematic weaknesses.
Researchers should assess risk of bias, heterogeneity, intervention definitions, and the appropriateness of combining different designs.
In some circumstances, structured narrative synthesis or triangulation may be more informative than a single pooled estimate.
Should Randomized Evidence Always Override Observational Evidence?
Randomized evidence often provides important advantages for causal identification, but interpretation should still consider the intervention, estimand, implementation, precision, and applicability.
A poorly implemented randomized trial may provide limited information about sustained intervention use.
A carefully designed natural experiment may address a related causal question under credible assumptions.
However, observational evidence should not be treated as equivalent to randomization merely because it produces a similar estimate.
Researchers should evaluate each study's identification strategy and limitations rather than rank evidence solely by design labels.
The purpose of triangulation is critical comparison, not automatic equivalence among methods.
How Can Researchers Plan Cross-Design Replication?
Begin by specifying the causal proposition that the research program seeks to investigate.
Then identify the most consequential alternative explanations and select designs that address them through different mechanisms.
For example, an observational longitudinal study may examine routine exposure patterns, while a randomized trial evaluates assignment to a specified intervention.
A policy-based quasi-experiment may provide additional evidence under a different assignment process.
Researchers should document where the designs share assumptions and where their vulnerabilities differ.
The strongest combination is not necessarily the largest number of studies. It is the set of credible studies that meaningfully reduces uncertainty about the proposed causal explanation.
How Should Researchers Evaluate Convergent Findings?
Convergence should be assessed across several dimensions.
Researchers should determine whether the studies address comparable causal questions, whether the estimates are compatible with their uncertainty, and whether differences in populations or interventions explain variation.
They should also evaluate whether the principal sources of bias are independent.
For example, a randomized trial and a regression discontinuity study may provide complementary evidence if their assumptions are credible and the estimated effects concern sufficiently related interventions.
However, agreement between several observational studies sharing the same unmeasured confounder may offer less additional causal support.
The evaluation should focus on the structure of the evidence rather than the number of statistically significant findings.
What If Cross-Design Replication Is Not Feasible?
Researchers may not have the resources, access, or ethical opportunity to conduct several designs.
A single well-designed study can still provide valuable causal evidence within its limitations.
Researchers should prioritize the design most appropriate to the causal question and the strongest plausible alternative explanations.
Where possible, they may use existing evidence from other studies to contextualize their findings.
However, a lack of cross-design replication should not be concealed, nor should researchers imply that one study has independently established every aspect of a causal proposition.
Replication is a strategy for strengthening cumulative evidence, not a universal prerequisite for every defensible causal estimate.
How Does Cross-Design Replication Relate to Alternative Explanations?
Its central value lies in testing whether the same causal interpretation remains plausible when different designs address different threats.
For example, a randomized trial may reduce concern about baseline selection, while a credible policy discontinuity may provide evidence from another assignment mechanism.
If both support compatible conclusions, certain alternative explanations become less persuasive.
However, other explanations may remain, particularly those shared across studies.
Researchers should therefore design causal studies around their strongest alternative explanations rather than assume that replication alone resolves them.
How Should Cross-Design Replication Be Reported?
Researchers should identify the causal proposition, intervention definitions, target populations, outcomes, and estimands examined across studies.
They should describe the identification strategy and principal assumptions of each design.
Convergent and conflicting findings should both be reported, including uncertainty and plausible explanations for differences.
Where studies share participants, datasets, instruments, or analytical assumptions, those dependencies should be disclosed.
The discussion should distinguish evidence convergence from definitive causal proof.
Researchers should avoid presenting several related analyses of the same dataset as fully independent replications.
Watch Out
Do not count studies as independent causal confirmations merely because they use different statistical methods or produce similar p-values. Cross-design replication is most informative when the designs address a comparable causal proposition through genuinely different identification strategies and sources of potential bias.
