03 · What You Need to Know
Combining Expert Review and Target-Population Evidence in Questionnaire Translation
Why Are Two Types of Review Necessary?
Translation quality involves both the intended meaning of the instrument and the meaning respondents actually construct when answering it.
Bilingual experts can compare source and target-language expressions, identify conceptual discrepancies, and examine whether technical terms have been translated appropriately.
Target-population members can identify unfamiliar wording, unexpected interpretations, and difficulties selecting responses.
These perspectives address related but different questions.
An item can be linguistically accurate but difficult for respondents to understand. Conversely, an item can be easy to understand while measuring a different construct from the original.
Combining the perspectives helps identify both problems.
What Should Bilingual Experts Evaluate?
Bilingual reviewers should examine whether the translated instrument preserves the intended conceptual and linguistic meaning.
Their evaluation may address specialized terminology, grammatical structure, idioms, negation, reference periods, instructions, and response categories.
For example, a bilingual reviewer familiar with educational psychology may identify that a translated self-efficacy item emphasizes motivation rather than perceived capability.
Another reviewer may identify an unnatural expression that could confuse respondents.
However, expertise in two languages does not automatically establish expertise in the research construct or measurement methodology.
Reviewers should be selected according to the competencies required by the instrument.
What Should Target-Population Members Evaluate?
Target-population review investigates whether intended respondents understand the translated questionnaire and can complete its response tasks meaningfully.
For example, students may be asked to explain what a translated item means, describe the experiences they considered, and explain why they selected a particular response.
These explanations can reveal interpretations that differ from the research team's intentions.
However, respondents should not be expected to determine whether an item accurately represents a specialized theoretical construct without appropriate support.
Their contribution is primarily evidence about comprehension, relevance, and response processes.
How Do the Two Perspectives Differ?
| Review Concern |
Bilingual Experts |
Target-Population Members |
| Semantic accuracy |
Compare intended meanings across languages |
Reveal how the translated expression is understood |
| Conceptual equivalence |
Evaluate alignment with the theoretical construct |
Identify unexpected interpretations of the concept |
| Natural language use |
Assess linguistic appropriateness |
Identify unfamiliar or awkward expressions |
| Cultural relevance |
Examine contextual assumptions |
Describe whether the experiences are recognizable |
| Response categories |
Evaluate semantic distinctions and ordering |
Explain how categories are selected |
| Construct coverage |
Review alignment with the measurement framework |
Identify experiences that may be missing or inappropriate |
| Measurement properties |
Identify possible measurement concerns |
Provide response-process evidence |
Neither review replaces empirical evaluation of reliability, structure, or measurement comparability when those properties are relevant to the intended use.
Can Bilingual Experts Be Members of the Target Population?
Yes. These categories are not necessarily mutually exclusive.
For example, a bilingual university student may possess the characteristics of the intended respondents while also being capable of comparing English and Filipino wording.
However, the reviewer should not automatically be treated as representative of all respondents.
Advanced language proficiency or familiarity with research terminology may distinguish the reviewer from other members of the target population.
Researchers should therefore consider whether additional respondents with different relevant experiences are needed.
Should Expert Review Come Before Target-Population Testing?
Often, a staged process is practical.
Initial linguistic and conceptual review can identify obvious problems before respondents are asked to evaluate the translated questionnaire.
Target-population testing can then investigate whether the reviewed version communicates the intended meaning.
However, the process need not be strictly linear.
Findings from cognitive interviews may require another round of expert review, followed by additional respondent testing.
The International Test Commission and ISPOR adaptation guidance support systematic attention to both expert review and target-language comprehension, although the precise procedures depend on the instrument.
How Many Bilingual Experts Should Participate?
There is no universal number that guarantees translation quality.
Some established adaptation procedures use multiple translators and a review committee, but their configurations differ.
The appropriate composition depends on the instrument's conceptual complexity, language pair, and intended use.
A review group should collectively provide the necessary linguistic, disciplinary, and measurement expertise.
Adding reviewers does not automatically improve the process if their competencies are redundant or important perspectives remain absent.
Researchers should justify the composition and document the relevant qualifications.
How Many Target-Population Members Are Needed?
No single sample size is appropriate for every translation review.
The number depends on the instrument's complexity, diversity of the intended respondents, evaluation method, and problems identified during testing.
For example, a brief questionnaire with straightforward items may require less extensive cognitive testing than a long instrument containing abstract constructs and complex response categories.
Researchers should consider iterative testing, particularly when substantial revisions occur.
A small number of interviews may identify important problems, but it cannot establish that all possible interpretations have been examined.
The adequacy of the review should be justified by the evidence obtained rather than a universal numerical threshold.
Should Target-Population Reviewers Represent Different Participant Characteristics?
Yes, when those characteristics could plausibly affect comprehension or response processes.
For example, a translated questionnaire intended for university students may be interpreted differently by first-year and graduating students if it contains specialized academic terminology.
Language proficiency, educational experience, accessibility needs, and familiarity with the measured activities may also be relevant.
Researchers should purposively consider these differences during cognitive testing.
However, qualitative pretesting is not necessarily intended to produce a statistically representative sample.
Its purpose is to investigate potential response problems and understand how they arise.
Is Asking Respondents Whether the Translation Is Clear Sufficient?
Not generally. Respondents may describe an item as clear while interpreting it differently from the intended construct.
For example, a student may find a translated question about self-regulated learning easy to read but interpret it as asking whether they study without supervision.
A simple clarity rating may not reveal that conceptual shift.
Researchers should investigate what respondents believe the question means and how they select their answers.
This distinction is central to cognitive interviewing.
How Can Cognitive Interviewing Improve Translation Review?
Cognitive interviewing examines the processes respondents use when answering questions.
Researchers may ask participants to explain an item in their own words, describe which experiences they considered, or explain how they distinguished adjacent response categories.
For example, a participant may interpret "I monitor my learning progress" as checking grades rather than evaluating understanding while studying.
That interpretation suggests a possible mismatch between the intended construct and the translated wording.
The research team can then revise the item and investigate whether the new wording improves comprehension.
Cognitive interviewing provides response-process evidence, but it does not independently establish the instrument's statistical measurement properties.
Should Participants Review the Original and Translated Versions?
It depends on the evaluation purpose.
Bilingual participants may be able to compare the two versions and identify perceived differences in meaning.
However, monolingual or less bilingual respondents may provide more informative evidence about how the translated instrument functions independently.
Showing respondents the original version can also influence how they interpret the translation.
Researchers should distinguish comparative bilingual review from independent target-language comprehension testing.
Both may be useful, but they answer different questions.
What If Experts Approve an Item but Respondents Misunderstand It?
The disagreement should prompt further investigation.
Expert approval indicates that reviewers considered the wording appropriate according to their criteria.
However, respondent misunderstanding suggests that the intended response process may not be occurring.
For example, a technically accurate translation of "metacognitive monitoring" may be unfamiliar to students.
The team should examine whether a more accessible expression can preserve the intended concept.
After revision, both conceptual review and respondent testing may be necessary.
Expert agreement should not override evidence of systematic misunderstanding merely because the reviewers have stronger academic credentials.
What If Respondents Prefer Wording That Experts Consider Inaccurate?
Preference alone should not determine the final translation.
Respondents may prefer familiar expressions that are easier to read but represent a different attribute.
For example, a phrase referring to general academic confidence may be preferred over one specifically describing perceived capability to complete difficult tasks.
The research team should investigate whether alternative wording can satisfy both conceptual accuracy and comprehension.
When necessary, several candidate expressions can be tested.
The decision should be grounded in the intended construct and observed response processes rather than popularity alone.
Can Expert Agreement Be Quantified?
Yes. Researchers sometimes calculate indices summarizing expert judgments about item relevance, clarity, or other criteria.
For example, content validity indices may summarize how reviewers rate particular items.
However, numerical agreement does not independently establish that translated items are conceptually equivalent or correctly understood by respondents.
Expert ratings also depend on the criteria, reviewers, and procedures used.
Researchers should report how the judgments were obtained and interpreted rather than treating a coefficient as comprehensive validation.
Does Target-Population Agreement Establish Validity?
No. Respondents may agree that an item is understandable while interpreting it consistently in an unintended way.
For example, all participants might interpret a translated autonomy item as asking whether they avoid consulting others.
The consistency of that interpretation would not establish that the item measures the intended theoretical definition of autonomy.
Target-population evidence should therefore be interpreted alongside conceptual review.
Validity requires an integrated argument supporting the intended score interpretation.
Can Target-Population Review Reveal Cultural Problems?
Yes. Respondents may identify activities, examples, or assumptions that do not correspond to their experiences.
For example, an instrument about academic engagement may refer to office-hour consultations that are organized differently in the target institution.
Respondents may understand the translated expression but have no equivalent experience to report.
Such findings indicate a potential problem of experiential relevance rather than linguistic accuracy alone.
The broader principles for adapting research instruments across cultural contexts are relevant when translation reveals culturally embedded assumptions.
Should Review Include Response Categories and Instructions?
Yes. The response task includes more than item wording.
Translated instructions may change the intended recall period, reference context, or basis for answering.
Similarly, response labels such as "Rarely," "Sometimes," and "Often" may not communicate sufficiently distinct frequencies in the target language.
Bilingual reviewers can examine semantic correspondence, while respondents can explain how they distinguish the categories.
Researchers should also verify scoring procedures when translation changes the wording or direction of items.
The review should cover the complete instrument.
Can AI Assist With Expert or Target-Population Review?
AI systems may help identify candidate wording, generate questions for reviewers, or organize non-sensitive feedback when permitted.
However, automated assessments of translation quality cannot substitute for qualified conceptual review or evidence from actual intended respondents.
An AI system may judge a phrase linguistically fluent while overlooking a construct shift or contextual problem.
Researchers should also consider licensing, confidentiality, and data-handling requirements before submitting instrument content or participant feedback to external systems.
AI assistance should be documented where relevant and should not be represented as independent human expert or participant evidence.
Do Both Review Groups Establish Measurement Equivalence?
Not independently. Expert review and target-population testing contribute evidence about conceptual relevance and response processes.
However, cross-language score comparisons may require additional statistical evidence.
For example, researchers comparing English and Filipino versions may need to investigate whether the measurement model operates comparably across language groups.
Depending on the instrument, relevant methods may include measurement invariance or differential item functioning analyses.
The necessary evidence depends on the intended comparison and measurement model.
Neither expert agreement nor respondent comprehension alone establishes numerical score interchangeability.
When Can a Less Extensive Review Be Defensible?
The appropriate evaluation depends on the complexity and consequences of the translation.
For example, a straightforward factual question may require less extensive review than a multidimensional psychological scale used as the primary outcome of an intervention study.
Similarly, an existing authorized translation with relevant supporting evidence may require targeted evaluation rather than complete redevelopment.
However, researchers should not assume that the presence of a previously translated version eliminates all need to examine population suitability.
The evaluation should be proportionate to the uncertainty associated with the intended use.
How Should Disagreements and Revisions Be Documented?
Maintain an item-level record identifying the original wording, translated wording, reviewer concerns, respondent interpretations, revisions, and final decisions.
For example, an item may be revised because experts identified a conceptual ambiguity that was subsequently confirmed during cognitive interviews.
Another item may be retained despite a different back-translation because respondents consistently understood its intended meaning.
Documentation should distinguish observed evidence from researcher interpretation.
The principles for documenting research instrument modifications support transparency and reproducibility.
How Should the Combined Review Process Be Reported?
Reports should identify the original instrument, target language, intended population, and review procedures.
Describe the relevant qualifications of bilingual experts, the characteristics of target-population participants, and the criteria used for evaluation.
Explain how consequential problems were identified and resolved.
Where cognitive interviewing was used, describe the procedures and how the findings informed revisions.
Report subsequent measurement evaluation separately from linguistic and respondent review.
A statement that the translation was "validated by experts and respondents" is less informative than a clear account of what each group evaluated and what the evidence supports.
Watch Out
Do not treat expert approval and respondent satisfaction as interchangeable evidence. Bilingual experts may establish that wording appears conceptually appropriate, while respondents may reveal that it is misunderstood. Conversely, easily understood wording may represent the wrong construct. Both perspectives must be interpreted together.
