What Should You Do When No Existing Measure Is Appropriate for Your Study Population?

What If None of the Available Research Measures Fits Your Participants?

You have reviewed the literature and identified several instruments for your research construct. Some have been used extensively, while others have promising psychometric evidence. Yet none appears fully appropriate for the people you intend to study.

Perhaps the available questionnaires were developed for adults, but your participants are adolescents. Their items may assume experiences that your population does not have, require unfamiliar terminology, or overlook dimensions particularly relevant to your setting.

Should you select the closest available instrument, modify an established measure, develop a new one, or reconsider your research question?

The decision matters because using an inappropriate instrument can produce scores that are numerically precise but conceptually misleading.

Investigate the Measurement Gap Before Deciding to Adapt or Develop an Instrument

How to Respond When Existing Measures Do Not Fit the Target Population

What Makes a Research Measure Inappropriate for a Particular Population?

A measure is inappropriate when its content, administration, scoring, or supporting evidence does not adequately justify the intended interpretation of scores for the population being studied.

Population suitability is not determined solely by age, nationality, educational attainment, or another demographic characteristic. The relevant issue is whether the instrument's measurement assumptions apply to the intended respondents.

For example, a questionnaire measuring workplace autonomy may include items about supervising employees, negotiating salaries, and selecting work schedules. Such items may be inappropriate for student interns who have never exercised those responsibilities.

Similarly, an assessment developed for highly literate adults may impose unnecessary reading demands when administered to younger respondents. In that case, the scores could partly reflect reading comprehension rather than the intended construct.

These problems concern more than convenience. They may affect construct representation, response processes, and the meaning of observed scores.

Why Doesn't Previous Validation Guarantee Suitability for Your Population?

Validity concerns the degree to which evidence and theory support particular interpretations of test scores for intended uses. It is not a permanent certification that transfers automatically to every new setting.

The Standards for Educational and Psychological Testing emphasize the relationship between validity evidence and intended score interpretation.

Evidence obtained from one population may be relevant to another, but researchers must examine whether important assumptions remain reasonable.

A scale developed among university students, for instance, may contain items that are understandable to secondary-school students. Nevertheless, differences in educational experiences, cognitive development, and institutional expectations could affect how respondents interpret those items.

The fact that a measure has been described as validated should therefore prompt examination of its supporting evidence rather than end the evaluation.

This is closely related to determining whether a measure is appropriate for the population and context, but the present problem requires deciding what to do when the available options do not adequately meet those requirements.

Is the Problem a Lack of Evidence or a Fundamental Mismatch?

One of the most important decisions is distinguishing an instrument that may be suitable but has not yet been evaluated in your population from one whose content is substantively inappropriate.

An evidence gap may sometimes be addressed through additional measurement evaluation without changing the instrument.

A substantive mismatch may require adaptation, replacement, or a different operational definition. Collecting additional reliability coefficients cannot resolve an instrument's failure to represent the intended construct adequately.

How Can You Diagnose the Specific Measurement Problem?

Review the available instruments at the level of their actual content and intended score interpretation. Do not rely exclusively on instrument names or descriptions in secondary publications.

Several types of mismatch may arise:

These problems may coexist. A measure can have both inadequate construct coverage and limited population-specific evidence.

Should You Adapt the Closest Existing Instrument?

Adaptation may be appropriate when an existing instrument provides a sound conceptual foundation and its limitations can be addressed without fundamentally changing the intended construct.

Potential adaptations include clarifying terminology, replacing context-specific examples, modifying administration procedures, or revising items that are inappropriate for the target population.

However, these changes should not be treated as automatically harmless.

Replacing an item about workplace decision-making with one about classroom decision-making, for example, may improve relevance but also alter the domain of autonomy being assessed.

The researcher must determine whether the modified instrument still represents the intended construct and whether the original scoring and measurement evidence remain applicable.

When considering adaptation of an existing instrument for a different population, the extent of modification and the supporting evidence should be proportionate to the intended use.

When Is Developing a New Instrument More Defensible?

New instrument development becomes more defensible when existing measures systematically omit essential aspects of the construct, contain fundamentally inappropriate assumptions, or cannot be adapted without substantially changing their meaning.

Development may also be warranted when the construct is genuinely population-specific and available instruments operationalize a different phenomenon.

For example, a researcher investigating self-regulation among learners using an unfamiliar educational delivery model may find that existing measures assume instructional activities that do not occur in the target setting.

If these assumptions are central to the items rather than incidental wording differences, creating a new instrument may provide a clearer measurement foundation.

Nevertheless, new development requires conceptual specification, item generation, content review, target-population involvement, and appropriate empirical evaluation.

The researcher should distinguish a genuine need for new measurement from a preference to create an original questionnaire.

The broader choice between using an existing measure and creating a new one should consider both the measurement gap and the resources needed to address it.

What Role Should Members of the Target Population Play?

Target-population involvement is particularly valuable when researchers are uncertain whether an instrument reflects participants' experiences or language.

Interviews, focus groups, cognitive interviewing, or other appropriate qualitative approaches can help identify relevant construct dimensions and reveal how respondents interpret proposed items.

For instance, an expert may consider a questionnaire item straightforward, while intended respondents interpret its terminology in several different ways.

Population involvement can therefore provide evidence about content relevance and response processes that expert review alone may not capture.

However, participants' preferences are not a substitute for theoretical justification or psychometric evaluation. An item may be easy to understand but still fail to measure the intended construct.

Can a Different Measurement Method Solve the Problem?

Sometimes the difficulty lies not in the questionnaire wording but in the measurement method itself.

A self-report questionnaire may be inappropriate for respondents who cannot reliably report the target behavior. Conversely, observational records may fail to capture private experiences, beliefs, or perceptions.

Depending on the construct, alternatives might include structured observations, performance assessments, interviews, administrative records, or carefully designed self-report procedures.

These methods provide different forms of evidence and should not be treated as interchangeable merely because they concern the same broad topic.

For example, observed participation in learning activities does not necessarily measure students' perceived engagement. A researcher should determine whether the alternative method addresses the original construct or requires a revised research question.

What Measurement Evidence Is Needed After Adaptation or Development?

The necessary evidence depends on the instrument and the claims the researcher intends to make.

Content validity is particularly important when population suitability is the central concern. The COSMIN framework emphasizes relevance, comprehensiveness, and comprehensibility as important aspects of content validity for patient-reported outcome measures. These distinctions are also informative for questionnaire development in other research areas.

Additional evidence may concern response processes, internal structure, relationships with other variables, reliability, and measurement error.

If the researcher intends to compare scores across populations, measurement invariance or another appropriate form of equivalence assessment may also be relevant.

The required evaluation should be planned before the main study where possible. Using an inadequately evaluated instrument as the primary outcome measure can create limitations that statistical analysis cannot subsequently eliminate.

Should Instrument Development Be Part of the Main Study?

Instrument development and substantive hypothesis testing are related but distinct research activities.

When an instrument is new or substantially modified, a preliminary development and evaluation phase may be necessary before it is used to answer the main research question.

Depending on the scope and resources, this work may form part of a multiphase project or be conducted as a separate study.

Using the same dataset to select items, establish a measurement model, and test substantive hypotheses can create risks of overfitting and overly optimistic conclusions, especially when decisions are driven by observed results.

Independent evaluation, separate samples, or carefully planned analytical procedures may help address these concerns.

When sufficient evaluation cannot be completed, the researcher should avoid presenting exploratory findings as though the instrument's measurement properties were already well established.

When Should You Reconsider the Research Question?

There are circumstances in which the available evidence and resources cannot support a defensible measurement strategy.

For example, a researcher may wish to measure a complex latent construct among a population for whom no appropriate instrument exists, while having insufficient time or access to participants for development and evaluation.

Proceeding with an arbitrary questionnaire may produce data but not a credible answer.

A narrower research question, qualitative exploratory design, or preliminary measurement-development study may be more appropriate.

Revising the study is not necessarily a methodological failure. It can be a recognition that the strength of a research conclusion depends on the quality of the evidence available to support it.

Measuring Self-Regulated Learning Among Students in a New Learning Environment

The appropriate response is not automatically to create a new questionnaire. The researcher must first establish whether the mismatch concerns a few context-specific indicators or a fundamental difference in how the construct is represented.

Common Mistakes When No Existing Measure Fits the Population

How to Decide Between Evaluation, Adaptation, New Development, and Redesign

Begin with a documented assessment of the measurement gap. Identify whether the difficulty concerns insufficient evidence, inappropriate wording, missing dimensions, administration barriers, or a fundamental mismatch between the instrument and the construct.

Then determine which response can realistically produce defensible measurements before the intended substantive analysis.

In the methodology section, explain the search for existing measures, the reasons they were considered unsuitable, and the rationale for the selected response.

Where adaptation or development is undertaken, describe the evidence supporting the resulting instrument and acknowledge any properties that remain insufficiently investigated.

A defensible measurement strategy does not require claiming that every uncertainty has been eliminated. It requires showing that the evidence is adequate for the conclusions the study intends to draw.

Before Concluding That No Suitable Measure Exists

Frequently Asked Questions About Population-Specific Measurement

Population Suitability Matters More Than Having an Established Instrument

How to Cite This Guide

This guide is intended to be read, shared, and used in research, teaching, and academic work. If you draw on its ideas, explanations, or other content, please acknowledge the source by citing the guide. Doing so gives appropriate credit and helps your readers locate the original resource.