03 · What You Need to Know
How to Determine Whether Local Adaptations Preserve Comparability
What Does Comparability Actually Require?
Comparability means that observations or estimates from different countries can be interpreted together for a specified scientific purpose. It does not necessarily mean that every country uses identical procedures or produces identical distributions.
Researchers should distinguish several forms of comparability. Conceptual comparability concerns whether a construct has sufficiently similar meaning. Measurement comparability concerns whether observations represent that construct on an appropriate scale. Population comparability concerns which people or institutions are represented. Intervention comparability concerns what was delivered or experienced.
Analytical comparability additionally requires that the intended estimates can be interpreted within a coherent statistical or qualitative framework.
A study may preserve one form of comparability while losing another. For example, two countries may investigate the same conceptual outcome using instruments whose scores cannot be compared directly.
Why Is There No Acceptable Percentage of Protocol Changes?
Protocol components do not contribute equally to scientific validity. Changing a meeting location is unlikely to have the same implications as changing the definition of the primary endpoint.
A numerical rule such as allowing 10% or 20% adaptation would treat these modifications as interchangeable. There is no general methodological basis for doing so.
The consequences also depend on the research question. A change in intervention delivery may be central to a study of implementation fidelity but relatively unimportant to a study examining a separate biological measurement.
Researchers should evaluate each modification according to its causal, measurement, sampling, and analytical implications rather than count amended protocol provisions.
Which Adaptations Are More Likely to Threaten Comparability?
| Adaptation |
Potential concern |
Evidence or response needed |
| Translated questionnaire |
Items may function differently across languages. |
Appropriate translation, response-process evaluation, and measurement evidence. |
| Different recruitment channels |
Different segments of the eligible population may participate. |
Coverage assessment, recruitment documentation, and selection-bias evaluation. |
| Different assessment modes |
Mode effects may change responses or task performance. |
Instrument-specific equivalence evidence or bridging assessment. |
| Modified intervention schedule |
Exposure timing or dose may change. |
Assessment of essential intervention components and implementation fidelity. |
| Different outcome instrument |
Scores may measure different constructs or lack a common scale. |
Validation, linking where defensible, or separate outcome interpretation. |
| Changed eligibility criteria |
Countries may represent different target populations. |
Reassessment of the estimand, selection, and intended comparisons. |
| Different follow-up periods |
Outcomes may represent different stages of change. |
Temporal alignment, suitable modeling, or restricted comparisons. |
These categories are not automatic judgments of acceptability. A translated instrument may require extensive evaluation, while a carefully calibrated alternative measurement system may provide suitable comparability.
How Does the Estimand Determine Acceptable Adaptation?
An estimand specifies the quantity the research intends to estimate. In intervention research, it may identify the target population, treatment conditions, outcome, handling of relevant post-assignment events, and population-level summary.
Consider a study estimating the effect of an instructional intervention on writing performance after eight weeks. If one country assesses performance after eight weeks and another after an entire academic term, the countries may no longer be estimating the same time-specific outcome.
Likewise, if one country evaluates the full intervention while another omits an essential component, the treatment conditions may differ.
The estimates may still be scientifically valuable, but their interpretation must reflect those differences.
Researchers should identify which adaptations change the estimand and which merely alter how an otherwise common estimand is operationalized.
When Does Measurement Adaptation Become Incompatible With Direct Comparison?
Measurement adaptation becomes problematic when the resulting scores no longer support the intended common interpretation.
For instance, replacing a performance-based writing assessment with a self-report questionnaire about writing confidence changes the construct. Even if both instruments use a 100-point scale, their scores cannot be treated as interchangeable measures of writing performance.
Other differences are less obvious. Two language versions may appear equivalent but exhibit differential item functioning or different factor structures.
Researchers may evaluate measurement invariance, agreement, calibration, or other relevant measurement properties depending on the instruments and intended comparisons.
The need to examine cross-cultural measurement equivalence is particularly important when comparing latent constructs or mean scores across populations.
Where sufficient equivalence cannot be established, direct comparison of raw scores may be inappropriate even if the broader research question remains shared.
Can Statistical Harmonization Restore Comparability?
Harmonization involves making data collected through different procedures sufficiently consistent for a defined analytical purpose.
Prospective harmonization establishes compatible definitions and procedures before data collection. Retrospective harmonization attempts to reconcile data that have already been collected.
Possible techniques include recoding compatible categories, converting units, linking instrument scores, and constructing common variables from appropriately comparable information.
However, harmonization cannot create information that was never collected or establish equivalence between fundamentally different constructs.
For example, converting examination scores into percentages does not establish that examinations of different difficulty or content measure equivalent achievement.
Researchers should document the assumptions, information loss, and uncertainty associated with harmonized variables. The Maelstrom Research guidance provides a structured approach to assessing the feasibility and quality of retrospective harmonization.
Can Different Sampling Procedures Still Produce Comparable Estimates?
Potentially, provided the target populations, sampling mechanisms, and analytical methods support the intended inference.
Different countries may use distinct sampling frames because their administrative systems differ. This does not automatically prevent comparison, but differences in population coverage and selection probabilities must be understood.
A probability sample from a national register and a convenience sample recruited through social media generally do not support the same population inference without substantial additional assumptions.
Statistical weighting may address known differences in selection probabilities or population composition under appropriate conditions. It cannot automatically eliminate selection bias or recover groups entirely absent from the data.
The decision about sampling strategies across countries should therefore precede decisions about pooling.
How Much Intervention Adaptation Can Be Accepted?
Intervention adaptation should be evaluated according to the functions and mechanisms that the intervention is intended to preserve.
Some interventions permit considerable flexibility in delivery while retaining essential components. Others depend strongly on dosage, timing, sequence, or standardized technical procedures.
For example, adapting culturally specific instructional examples may preserve an educational intervention's core feedback mechanism. Removing feedback entirely would change the intervention substantially.
Frameworks such as FRAME can help document what was modified, why, when, and by whom. They do not, by themselves, establish that adapted interventions are equivalent.
Researchers should evaluate whether different intervention versions remain relevant to the common estimand or require explicit treatment-version distinctions.
What Happens When Adaptation Is Confounded With Country?
A particularly difficult situation occurs when each country uses a different procedure and no country uses more than one version.
Suppose Country A administers an assessment online, while Country B administers it on paper. If scores differ, the difference may reflect national context, assessment mode, participant composition, or some combination.
Because mode and country coincide, the data may not identify their separate effects without additional assumptions or bridging evidence.
A bridging study in which both modes are used within suitable populations can sometimes provide relevant information. Its usefulness depends on the design and whether the resulting calibration relationship transfers to the intended populations.
Simply including country as a regression covariate does not necessarily resolve this identification problem.
When Should Data Be Pooled, Stratified, or Analyzed Separately?
Pooling may be appropriate when a common estimand is meaningful and the data support the necessary measurement, population, and analytical assumptions.
Stratified or country-specific analyses may be preferable when differences are scientifically important, when effect heterogeneity is central to the question, or when a common estimate would conceal incompatible interpretations.
Separate analyses may be necessary when countries measure different outcomes or implement interventions that cannot reasonably be treated as versions of the same treatment condition.
Even when pooling is defensible, researchers should consider whether country-specific analyses are needed to understand important contextual differences.
Decisions should be prespecified where feasible and revisited transparently when unexpected comparability problems emerge.
How Should Adaptation Decisions Be Governed?
A multicountry study benefits from a documented process that distinguishes minor operational adaptations from changes affecting scientific comparability.
Local teams should explain the contextual need for a proposed modification. The coordinating team and relevant methodological specialists should assess its implications for the research question, measurement, sampling, intervention, and analysis.
Changes requiring ethics, regulatory, or institutional authorization must receive the necessary approvals before implementation, subject to applicable rules.
Documentation should identify the original procedure, modified procedure, rationale, evidence, approval, implementation date, and affected observations.
This governance process should complement the standardized scientific core of the multicountry protocol.
How Can Researchers Assess Comparability Before Data Collection?
Prospective assessment is generally preferable to discovering incompatibilities after data collection.
Researchers can map proposed national procedures against the common protocol, identify consequential differences, review existing validation evidence, and conduct targeted pilot or bridging studies.
They should prioritize adaptations affecting primary outcomes, treatment conditions, eligibility, and the target population.
A comparability matrix can record the proposed procedure in each country, the scientific purpose it serves, supporting evidence, and the decision about whether observations can be combined.
When evidence is insufficient, investigators should specify the remaining uncertainty and determine whether additional validation is feasible.
Watch Out
Do not interpret the number of approved adaptations as evidence of comparability. A study can preserve nearly every procedural detail yet lose comparability through one consequential change to its primary outcome or treatment condition.
