03 · What You Need to Know
Why Target-Population Involvement Matters in Instrument Development
What Is Target-Population Review?
Target-population review involves obtaining structured feedback from people who resemble the intended respondents or users of a research instrument.
The purpose is to examine whether the proposed content is relevant to their experiences, whether instructions and items are comprehensible, and whether the response tasks can be completed as intended.
For example, students may review a questionnaire designed to measure academic help-seeking, while practicing teachers may evaluate items concerning instructional decision-making.
The participants' role differs from that of subject-matter experts. They contribute evidence grounded in the experiences and response processes of the population for whom the instrument is intended.
Target-population review can be conducted through individual interviews, structured feedback sessions, cognitive interviewing, or other suitable procedures.
It should not be confused with merely asking a few convenient acquaintances whether a questionnaire looks acceptable.
Why Is Expert Review Alone Insufficient?
Experts can evaluate theoretical relevance, technical accuracy, and conceptual coverage. However, their disciplinary knowledge may influence how they interpret item wording.
A technical expression that is familiar to an expert may be unfamiliar or ambiguous to intended respondents.
For example, an educational researcher may interpret "metacognitive monitoring" as a defined process involving awareness and evaluation of one's understanding. A student may interpret the same phrase as simply paying attention during class.
Both individuals can understand the words, but they may assign different meanings to the statement.
Target-population review can reveal such differences before they affect the main dataset.
The distinction complements expert review of instrument items before pilot testing. Both forms of evaluation are useful because they address different aspects of measurement quality.
What Can Members of the Target Population Evaluate?
Intended respondents can provide evidence about several features of the proposed instrument.
| Evaluation Area |
Question to Investigate |
Illustrative Problem |
| Comprehensibility |
Do respondents understand the item as intended? |
A technical term has an unfamiliar meaning |
| Relevance |
Does the item concern experiences applicable to respondents? |
A question assumes access to equipment not used by participants |
| Recall |
Can respondents retrieve the information requested? |
A question asks for an exact frequency over an unrealistic recall period |
| Judgment |
Can respondents meaningfully evaluate the attribute? |
An item asks students to assess competence they cannot independently verify |
| Response options |
Can respondents express an appropriate answer? |
No category represents a legitimate response |
| Acceptability |
Is the item appropriate and reasonably comfortable to answer? |
A sensitive question appears unnecessarily intrusive |
| Administration |
Can respondents navigate the instrument as intended? |
Mobile presentation obscures response labels |
These evaluations should be connected to the intended measurement. A participant's preference for shorter wording, for example, is useful feedback but does not automatically establish that the shorter version is conceptually superior.
How Does Target-Population Review Contribute to Content Validity?
Content validity concerns whether the content of an instrument adequately represents the intended construct for a specified purpose.
In the COSMIN methodology for patient-reported outcome measures, relevance, comprehensiveness, and comprehensibility are central considerations.
Members of the target population can contribute evidence to each of these areas.
They may identify content that does not reflect their experiences, describe important aspects omitted from the questionnaire, or reveal wording that is interpreted differently from the researcher's intention.
However, target-population review does not independently establish the entire validity argument.
Participants may find an item relevant and easy to answer even when it measures an attribute outside the intended construct. Theoretical and measurement expertise remains necessary.
What Is the Difference Between Target-Population Review and Pilot Testing?
Target-population review generally focuses on examining the proposed instrument's content and respondents' experiences with its questions.
Pilot testing more broadly examines whether the instrument and data collection procedures function under conditions resembling the main study.
Target-population review
Investigates whether intended respondents find the instrument relevant, understandable, answerable, and appropriate.
Pilot testing
Evaluates the practical functioning of an instrument or data collection process, potentially including completion, recording, timing, and preliminary measurement behavior.
These activities can overlap. For example, a pilot study may incorporate cognitive interviews or structured participant feedback.
The distinction concerns the questions being investigated rather than a mandatory separation into different studies.
Who Should Participate in the Review?
Review participants should resemble the intended population in characteristics relevant to understanding and answering the instrument.
These characteristics may include educational level, language proficiency, age, professional experience, disability-related access needs, or familiarity with the topic.
For example, a questionnaire intended for first-year university students should not be reviewed exclusively by postgraduate researchers who are already familiar with the terminology.
Likewise, an instrument intended for teachers across different school settings may require reviewers representing different teaching contexts.
The purpose is not necessarily statistical representativeness. In qualitative pretesting, purposive recruitment may be appropriate for examining variation in response processes.
However, a convenience group that excludes important segments of the intended population may fail to reveal consequential problems.
How Many Target-Population Members Should Review the Items?
There is no universally sufficient number.
The appropriate number depends on the review method, diversity of the population, complexity of the instrument, and purpose of the investigation.
Qualitative cognitive interviews may use iterative rounds with relatively small groups, allowing researchers to revise items and investigate whether problems persist.
A broader quantitative pilot may require substantially more participants to estimate particular measurement properties with adequate precision.
Researchers should therefore avoid using a fixed number of respondents as evidence that an instrument has been adequately reviewed.
Instead, explain the sampling rationale, the problems the review was designed to detect, and whether additional review was necessary.
What Is Cognitive Interviewing?
Cognitive interviewing is a method for investigating how respondents understand, retrieve information for, judge, and answer survey questions.
It may involve asking respondents to explain what a question means to them, describe how they selected an answer, or identify difficulties encountered during the response process.
For example, a respondent might be asked what the phrase "regularly evaluate academic sources" means and how they decided which frequency category to select.
These explanations can reveal whether respondents are answering the question the researcher intended.
Cognitive interviewing should be conducted systematically, with attention to interviewer prompts, interpretation, and documentation.
The detailed procedures for cognitive interviewing in questionnaire development require separate methodological consideration.
Should Participants Think Aloud While Answering?
Think-aloud procedures ask respondents to verbalize aspects of their thought process while completing a task.
They can help researchers identify uncertainty, unfamiliar terminology, or difficulties selecting responses.
However, thinking aloud may be unfamiliar or demanding for some respondents. Verbalization can also influence how a person approaches the question.
Another approach uses retrospective probing, in which respondents answer an item and then explain how they interpreted it.
These approaches are not interchangeable in every context. The appropriate method depends on the instrument, population, and information required.
Researchers should avoid assuming that respondents can provide complete or perfectly accurate accounts of every cognitive process involved in answering.
Should Participants Be Asked Whether They Like the Questions?
General satisfaction can provide information about acceptability, but it is insufficient for evaluating measurement quality.
A respondent may say that every item is clear while interpreting several questions differently from the intended meaning.
More informative questions investigate specific response processes.
For example, researchers might ask respondents to explain a key term in their own words, describe what experiences they considered, or identify situations in which none of the response categories would fit.
Such questions can reveal problems that a simple request for approval would miss.
The objective is not merely to make respondents comfortable with the questionnaire. It is to determine whether their responses can support the intended interpretation.
Can Target-Population Members Identify Missing Construct Content?
Yes. Intended respondents may describe relevant experiences that were not represented in the initial item pool.
For example, students reviewing a self-regulated learning questionnaire for asynchronous education might identify the importance of managing interruptions at home or coordinating learning around irregular schedules.
These experiences could suggest additional item content.
However, the researcher must determine whether the proposed content belongs within the construct or represents a contextual influence, antecedent, or consequence.
Participant suggestions should therefore be interpreted through the conceptual framework rather than automatically incorporated into the instrument.
This distinction is particularly important when the instrument measures a narrowly defined construct.
Can Participants Identify Problems With Response Options?
Yes. Response categories may be difficult to interpret, insufficiently differentiated, or unsuitable for the requested judgment.
For example, a questionnaire may ask how often students use a particular learning strategy but offer only agreement categories.
Another item may assume that all participants have performed an activity, leaving no appropriate option for those without relevant experience.
Respondents may also interpret labels such as rarely or frequently differently.
Target-population review can identify these difficulties, although the final response format should also reflect the measurement model and intended analysis.
The separate decision about how many response options a questionnaire item should have should be based on the judgment requested and the evidence supporting the response process.
How Should Researchers Handle Conflicting Participant Feedback?
Different participants may interpret an item differently or prefer different formulations.
Such disagreement can be informative, particularly when it reflects differences in language, educational experience, or familiarity with the topic.
Researchers should investigate whether the disagreement indicates genuine ambiguity, population heterogeneity, or simply individual preferences.
For example, one group may understand "learning module" as a complete unit of instruction, while another understands it as a single digital resource.
If the instrument is intended for both groups, the terminology may need clarification.
However, not every preference should lead to a revision. Changes should be evaluated against the construct definition and the needs of the full target population.
Should Participants Have the Final Authority Over Item Wording?
Not necessarily. Participant feedback is essential for understanding how items function from the respondent's perspective, but instrument development requires integrating several kinds of evidence.
Participants may prefer familiar language that is conceptually imprecise. Experts may prefer technical language that respondents cannot understand consistently.
The researcher's responsibility is to reconcile these concerns through defensible wording and further testing where necessary.
For example, a technically accurate term may be replaced with a more accessible expression if the revised wording preserves the intended meaning.
The decision should not be framed as choosing whether experts or participants are more authoritative in general. Their expertise concerns different aspects of the measurement process.
When Should Target-Population Review Occur?
Review is often useful after the initial construct definition, item generation, and preliminary content evaluation, but before the instrument is finalized for primary data collection.
However, target-population involvement can also contribute earlier in development, particularly when the construct is emerging or poorly understood.
For example, qualitative interviews may help establish the content domain before any items are written.
Later cognitive interviews may evaluate proposed wording, while pilot testing examines how the instrument functions in practice.
There is no single mandatory sequence for every instrument. The stages should be organized according to the evidence needed and the consequences of making revisions.
Should Revised Items Be Reviewed Again?
Revised items may require additional review when changes affect their meaning, response task, or contextual relevance.
For example, replacing a technical term with a simpler expression may resolve one misunderstanding while introducing another.
Iterative review allows researchers to examine whether revisions actually address the identified problem.
However, repeated review should have a clear purpose. It is not necessary to continue indefinitely until every participant expresses complete satisfaction.
Researchers should document the revision history and determine when sufficient evidence has been obtained to proceed to the next development stage.
What Ethical Considerations Apply to Target-Population Review?
Participants should understand the purpose of the review and what participation involves.
Appropriate consent, privacy protections, and institutional ethics requirements should be considered according to the nature of the activity and applicable rules.
Researchers should be especially attentive when reviewers are students, employees, patients, or others who may perceive pressure to participate because of an existing relationship.
For example, students should not feel that their grades or standing depend on providing favorable feedback about a questionnaire developed by their instructor.
Review procedures should also accommodate relevant accessibility and language needs.
When feedback sessions are recorded, the purpose, handling, and retention of recordings should be communicated appropriately.
How Should Target-Population Feedback Be Documented?
Researchers should maintain records linking identified problems to decisions about item revision.
A useful record may include the item identifier, respondent characteristics relevant to interpretation, observed difficulty, supporting evidence, proposed revision, and subsequent evaluation.
For example, a review log might record that several students interpreted "independent learning" as studying without any instructor support, although the intended meaning concerned regulating their own learning activities.
The research team could then revise the item and examine whether the new wording better communicates the intended attribute.
Documentation should preserve confidentiality and avoid unnecessary collection of identifiable information.
Transparent records help distinguish evidence-based revisions from changes made solely according to researcher preference.
Does Target-Population Review Establish That an Instrument Is Valid?
No. It contributes evidence relevant to the instrument's content and response processes.
Further evaluation may be needed to examine dimensional structure, reliability, measurement error, relationships with other variables, and comparability across groups.
For example, students may understand every questionnaire item correctly, but the resulting responses may not support the proposed factor structure.
Conversely, an instrument may produce high internal consistency while respondents interpret important items inconsistently.
The Standards for Educational and Psychological Testing emphasize that validity concerns the evidence supporting interpretations of scores for particular uses.
Target-population review is therefore part of a broader validity argument rather than a one-time certification.
Watch Out
Do not conclude that questionnaire items are appropriate simply because intended respondents say they are clear or easy to answer. Comprehensibility requires evidence about how questions are actually interpreted, and conceptual relevance must still be evaluated against the intended construct.
