03 · What You Need to Know
How to Determine Whether New Instrument Development Is Necessary
What Counts as Developing a New Research Instrument?
Research instrument development involves systematically creating and evaluating a procedure for obtaining observations or scores intended to represent a defined construct or quantity.
In questionnaire research, this may involve conceptualizing the construct, identifying its dimensions, generating items, designing response options, establishing scoring procedures, and evaluating measurement properties.
Other instruments may require different activities. Developing a performance assessment involves specifying tasks and scoring criteria, while developing an observational instrument may require defining observable behaviors and establishing coding procedures.
The central requirement is not merely producing questions or recording categories. It is establishing a defensible relationship between the observations and the interpretations the researcher intends to make.
The Standards for Educational and Psychological Testing emphasize that validity concerns evidence and theory supporting particular interpretations of scores for intended uses. This principle applies to newly developed instruments as well as established ones.
Why Should You Examine Existing Measures Before Developing a New One?
Existing instruments may already provide relevant content, established scoring procedures, and evidence concerning reliability, internal structure, and relationships with other variables.
Using an appropriate established instrument can reduce unnecessary development work and facilitate comparison with earlier studies.
However, the presence of a widely used measure does not establish its suitability for every research question.
Researchers should examine the actual items, construct definitions, target populations, administration requirements, and available validity evidence.
The decision should follow the distinction between using an existing measure and creating a new one, with particular attention to whether the identified limitation requires genuine instrument development.
A literature search should include original instrument-development studies, subsequent measurement evaluations, relevant reviews, and official instrument documentation where available.
What Constitutes a Genuine Measurement Gap?
A genuine measurement gap exists when available instruments cannot adequately support the intended measurement or score interpretation.
Such gaps may concern construct coverage, population suitability, measurement purpose, administration requirements, or the absence of defensible operational indicators.
| Potential Gap |
Illustration |
Possible Response |
| Missing construct dimensions |
Existing scales omit an essential aspect of the theoretically defined construct |
Consider adaptation or new development after examining the conceptual gap |
| Population mismatch |
Items assume experiences unavailable to the intended participants |
Evaluate population-specific adaptation or development |
| Different measurement purpose |
Available questionnaires assess perceived ability rather than demonstrated competence |
Consider an appropriate performance-based instrument |
| Inadequate measurement range |
Existing items provide insufficient information at the construct levels relevant to the study |
Investigate alternative instruments or targeted development |
| Contextual incompatibility |
Instrument content assumes institutional practices absent from the study setting |
Determine whether contextual adaptation preserves the construct |
| Insufficient existing evidence |
An otherwise suitable instrument has not been evaluated in the intended population |
Consider additional validation work before assuming a new instrument is necessary |
The final example is particularly important. A lack of population-specific evidence is not necessarily a reason to discard an existing instrument. The measure may be suitable but require additional evaluation.
When Is an Existing Instrument Better Than a New One?
An established instrument is generally attractive when its content matches the intended construct, its administration is feasible, and relevant evidence supports the proposed score interpretation.
It may also be preferable when the study seeks to replicate earlier findings or compare results across populations.
Developing a new instrument in these circumstances may introduce uncertainty without improving the measurement.
For example, replacing a well-supported academic self-efficacy scale with newly written items simply because the researcher prefers different wording may weaken comparability and require substantial additional evaluation.
The most widely used measure is not automatically the best measure, but neither is a newly developed instrument automatically superior to an established one.
When Is Adaptation Preferable to New Development?
Adaptation may be appropriate when an existing instrument has a suitable conceptual foundation but requires limited, justified changes for the intended population or setting.
Examples include modifying context-specific wording, preparing an appropriate language version, or adjusting administration procedures.
However, adaptation can change construct coverage, response processes, or scoring. Its consequences should be evaluated rather than assumed negligible.
When existing measures do not fit the study population, researchers should distinguish between an instrument that can be adapted responsibly and one whose underlying conceptual assumptions are fundamentally unsuitable.
If extensive modification would replace most items, redefine the dimensions, and require a substantially different scoring model, the resulting work may be better understood as new instrument development informed by an existing measure.
There is no universally accepted percentage of changed items that separates adaptation from development.
Does a New Research Construct Necessarily Require a New Instrument?
Not always. A newly proposed theoretical construct may overlap with established constructs or combine dimensions already measured by existing instruments.
Before developing items, researchers should examine whether the proposed construct is conceptually distinguishable from related concepts.
For example, a researcher proposing a measure of confidence in evaluating AI-generated academic information should investigate existing work on information literacy, critical evaluation, digital competence, and self-efficacy.
The relevant question is whether the proposed construct represents something sufficiently distinct to require different measurement indicators.
Simply assigning a new label to familiar questionnaire content does not establish conceptual or measurement novelty.
Where existing measures cover important dimensions, they may inform the conceptual analysis even if they cannot be adopted unchanged.
How Clearly Must the Construct Be Defined Before Development?
A defensible instrument requires a clear conceptual definition of what it is intended to measure.
The researcher should specify the construct's boundaries, relevant dimensions, and relationship to adjacent concepts.
Without this foundation, item development may become an exercise in collecting plausible statements rather than representing a coherent measurement domain.
For example, a proposed AI literacy instrument might unintentionally combine technical knowledge, confidence, ethical attitudes, and frequency of AI use without explaining whether these are dimensions of one construct or distinct attributes.
The measurement model should follow the conceptual framework rather than being inferred solely from whichever items produce convenient statistical results.
Conceptual clarity does not eliminate the need for empirical evaluation, but it provides the basis for deciding what the instrument should represent.
What Evidence Is Needed to Support a Newly Developed Instrument?
The necessary evidence depends on the instrument and intended use.
For questionnaire-based measures, development commonly involves examining content relevance, comprehensiveness, and comprehensibility, followed by suitable investigations of response processes and measurement properties.
Depending on the construct and scoring model, researchers may evaluate internal structure, reliability, measurement error, relations with other variables, and measurement invariance.
The COSMIN methodology provides a structured approach to evaluating measurement properties, particularly for health-related outcome instruments. Its distinctions can inform questionnaire evaluation in other disciplines when applied appropriately.
Boateng and colleagues describe a multistage approach to scale development involving item generation, scale construction, and scale evaluation.
These frameworks should not be treated as rigid universal sequences. Development procedures must be appropriate to the measurement method and intended inference.
Why Is Expert Review Not Enough?
Expert review can contribute evidence about construct relevance, content coverage, and item clarity.
However, experts cannot establish every aspect of how intended respondents will understand and answer the items.
Members of the target population may interpret terminology differently or identify relevant experiences that experts overlooked.
Furthermore, favorable expert ratings do not establish reliability, structural validity, or appropriate relationships with external variables.
Expert review is therefore one component of an evidence-building process, not a complete certification of validity.
Why Should the Target Population Be Involved?
Target-population involvement can help researchers determine whether the proposed instrument reflects the experiences and language of the people being studied.
Interviews, focus groups, cognitive interviewing, and other appropriate methods may identify missing content, misunderstood questions, or unsuitable response categories.
For example, students may interpret a proposed item about independent learning differently depending on whether their courses involve self-paced modules or scheduled instructor-led activities.
Such evidence can improve content relevance and comprehension before large-scale administration.
However, target-population feedback should be interpreted alongside theory and measurement evidence. Popular wording is not necessarily conceptually appropriate wording.
How Much Empirical Evaluation Is Necessary?
There is no universally sufficient sample size or single statistical test that establishes a new instrument's validity.
The required evaluation depends on the measurement model, number and type of items, intended precision, target population, and analytical objectives.
Exploratory analyses may help investigate dimensional structure, while confirmatory analyses can evaluate a prespecified model under appropriate assumptions.
Reliability evidence should be selected according to the intended score and measurement conditions. Internal consistency alone is not sufficient for every instrument.
When scores will support high-consequence decisions or detailed individual comparisons, stronger evidence may be necessary than for preliminary exploratory research.
Researchers should avoid claiming comprehensive validation after a single small pilot or one acceptable coefficient.
Should You Develop an Instrument Within the Same Study That Tests Your Hypotheses?
Instrument development and substantive hypothesis testing can occur within a coordinated research project, but their methodological purposes should remain distinguishable.
Developing and selecting items using the same dataset subsequently used to confirm the measurement model or test substantive relationships may capitalize on sample-specific characteristics.
Independent evaluation, separate samples, or carefully planned analytical strategies can reduce some of these concerns.
Where development is substantial, conducting instrument development as a separate study may be more defensible.
However, separation into different publications is not itself a guarantee of measurement quality. The important issue is whether development and evaluation are adequately designed and transparently reported.
What Resources Should Be Available Before Development Begins?
Instrument development requires more than time to write items.
Researchers may need subject-matter expertise, access to intended respondents, qualitative data collection, statistical expertise, suitable sample sizes, and resources for repeated refinement.
Some instruments also require equipment, specialist scoring, software, or permission to incorporate protected material from existing measures.
Development timelines should accommodate unexpected findings. Items may prove incomprehensible, the proposed dimensional structure may not be supported, or the resulting scores may lack sufficient precision.
Researchers should therefore assess whether they can complete the evidence-building process before relying on the instrument for the main study.
When Should You Avoid Developing a New Instrument?
New development is difficult to justify when an existing measure adequately addresses the research question and the proposed instrument offers no substantive improvement.
It may also be inappropriate when the research team lacks the resources to evaluate the resulting scores but intends to use them for strong confirmatory claims.
Other warning signs include unclear construct definitions, item generation based solely on intuition, and the assumption that expert approval or a high Cronbach's alpha completes validation.
In these circumstances, using an established instrument, undertaking preliminary exploratory research, or revising the study may provide a stronger methodological foundation.
Watch Out
Do not develop a new questionnaire merely to claim originality. Instrument development is justified by a defensible measurement need, and the resulting scores require evidence supporting their intended interpretation. New wording is not equivalent to a new construct, and a newly constructed scale is not automatically a validated instrument.
