03 · What You Need to Know
How to Decide Whether New Instrument Development Belongs in Your Research Design
Distinguish Instrument Development From Instrument Use
A research instrument is a structured means of obtaining observations or measurements. Depending on the discipline, it may take the form of a questionnaire, psychological scale, achievement test, observational rubric, interview protocol, performance assessment, or technical measurement procedure.
Not all instruments require the same development and evaluation procedures. A standardized scale intended to estimate a latent psychological construct has different measurement requirements from a factual demographic questionnaire or a qualitative interview guide.
For a scale intended to produce interpretable numerical scores, development commonly involves defining the construct, generating items, evaluating content, examining response processes, testing the scoring structure, and investigating reliability and validity evidence.
By contrast, a qualitative interview protocol may require careful alignment with the research question, methodological coherence, pilot interviewing, and reflexive evaluation rather than psychometric factor analysis.
The first decision is therefore to identify what kind of instrument is needed and what claims its outputs must support.
Why Is Instrument Development More Demanding Than Writing Questions?
Writing understandable questions is an important activity, but it is only one part of establishing a useful measurement instrument.
A questionnaire may appear relevant to its topic while producing scores influenced by ambiguous wording, inappropriate response options, cultural assumptions, or construct-irrelevant factors.
For example, a questionnaire intended to measure students' AI literacy might include items about confidence in using AI tools. Confidence may be relevant, but it does not necessarily demonstrate actual knowledge or performance.
If the intended construct includes the ability to evaluate AI-generated information critically, self-reported confidence alone may provide an incomplete representation.
Instrument development requires a defensible account of how observations or responses relate to the construct and why the proposed interpretation is appropriate.
When Is Developing a New Instrument Scientifically Justified?
A new instrument may be justified when existing measures cannot adequately address the intended measurement purpose.
Potential reasons include an inadequately represented construct, an emerging phenomenon, a substantially different population, or measurement requirements not addressed by available instruments.
However, the absence of an instrument with precisely the same title as the research topic is not sufficient justification.
Researchers should examine whether established instruments already measure the underlying construct, even when they use different terminology.
For example, a study of students' trust in AI-supported feedback might draw on established concepts from trust, technology acceptance, perceived credibility, or reliance. Whether an existing measure is suitable depends on its conceptual alignment and evidence for the intended use.
Developing a new scale solely because the study involves a new technological context may unnecessarily duplicate existing measurement work.
What Should You Do Before Deciding That No Suitable Instrument Exists?
Conduct a focused review of relevant measurement literature.
Search for instruments using the construct's established terminology, related theoretical concepts, and common measurement approaches. Examine original development studies, subsequent validation studies, and applications involving populations similar to your own.
For each potentially suitable instrument, evaluate:
- The construct and dimensions the instrument was designed to measure.
- The population and context in which it was developed.
- The intended scoring and interpretation.
- Available evidence concerning validity, reliability, and measurement structure.
- Language, cultural, and accessibility considerations.
- Licensing, permission, administration, and scoring requirements.
- Whether its content adequately represents the current research question.
The COSMIN initiative provides structured resources for evaluating measurement properties, particularly for health-related outcome instruments. Its concepts may inform measurement evaluation in other disciplines, although researchers should select standards appropriate to their own instrument and context.
Distinguish Using, Adapting, and Developing an Instrument
These options differ in the amount of new measurement work required.
| Approach |
What it involves |
Principal methodological concern |
| Use an existing instrument |
Administer an established measure according to appropriate procedures |
Suitability of existing evidence for the new population and purpose |
| Adapt an existing instrument |
Modify wording, language, context, format, or selected content |
Whether the adaptation changes the construct or measurement properties |
| Develop a new instrument |
Define the construct, create measurement content, and gather new validity evidence |
Whether the development and evaluation process is sufficiently comprehensive |
Adaptation is not automatically a minor task. Translating a scale, changing response categories, removing items, or altering the target population may affect score interpretation and comparability.
Even when the original instrument has strong validity evidence, its suitability for a new context must be assessed rather than assumed.
What Does a Defensible Instrument Development Process Involve?
The required stages depend on the type of instrument and its intended use. For a multi-item quantitative scale measuring a latent construct, a defensible process may involve several connected activities.
Define the construct and intended use.
Specify what the instrument is intended to measure, for whom, under what conditions, and how its scores will be interpreted.
Develop a conceptual framework.
Use relevant theory and empirical literature to identify the construct's boundaries and potential dimensions.
Generate and review items.
Develop items that represent the intended content and evaluate their relevance, clarity, and coverage.
Investigate response processes.
Use suitable techniques, such as cognitive interviewing or pilot administration, to examine how intended respondents understand and answer the items.
Evaluate measurement properties.
Investigate the structure, reliability, and relevant validity evidence using methods appropriate to the proposed scoring model.
Refine the instrument and document limitations.
Revise problematic items, evaluate the consequences of changes, and state what the available evidence does and does not establish.
These activities are not necessarily completed in one linear sequence. Development may require several iterations, and evidence supporting score interpretation may continue to accumulate after the initial study.
A thesis that includes instrument development should specify which stages can realistically be completed and what level of validation evidence the final instrument will possess.
Why Is Content Validity Important?
Content validity concerns the extent to which an instrument's content adequately represents the construct for its intended purpose.
For example, a scale intended to measure responsible generative AI use might need to distinguish relevant knowledge, practices, or judgments based on a defensible conceptual definition.
Expert review can help evaluate whether items are relevant and sufficiently comprehensive. Input from intended respondents may also be important for assessing comprehensibility and whether important experiences or meanings have been overlooked.
However, asking several experts to approve a questionnaire does not establish every aspect of validity.
Expert agreement cannot by itself demonstrate that respondents interpret items as intended, that a proposed scoring structure is appropriate, or that scores relate to other variables in theoretically expected ways.
Does Cronbach's Alpha Prove That an Instrument Is Valid?
No. Cronbach's alpha is commonly used as an indicator of internal consistency under particular assumptions. It does not establish that an instrument measures the intended construct.
A high alpha may arise from strongly related or redundant items. It does not independently establish unidimensionality, content validity, or appropriate score interpretation.
Internal consistency should be interpreted alongside the instrument's measurement model and other evidence.
Depending on the instrument, researchers may consider additional reliability estimates, test-retest evidence, inter-rater reliability, or other relevant measurement properties.
The choice should follow the intended use rather than an assumption that every instrument must report the same coefficient.
What Is the Difference Between Reliability and Validity?
Reliability
The consistency or precision of measurement under specified conditions, evaluated using methods appropriate to the instrument and intended interpretation.
Validity
The extent to which evidence and theory support the intended interpretation and use of measurements or scores.
Reliability is important because highly inconsistent measurement can undermine useful interpretation. Nevertheless, consistent scores may still represent the wrong construct.
For example, a questionnaire might consistently measure students' confidence in using AI tools while being incorrectly interpreted as a direct assessment of their AI knowledge.
Reliable measurement of confidence does not establish valid measurement of knowledge.
How Much Data Are Needed to Validate a New Instrument?
There is no universally appropriate sample size for every instrument validation study.
Requirements depend on the number of items, expected factor structure, strength of relationships among items, estimation method, population characteristics, and intended analyses.
For example, exploratory factor analysis and confirmatory factor analysis have different purposes, and their sample requirements depend on more than a simple participants-per-item ratio.
Simulation research by MacCallum and colleagues demonstrated that factor analysis sample requirements are influenced by characteristics such as communalities and factor overdetermination.
Rules suggesting that every questionnaire requires a fixed number of respondents or a universal ratio of participants to items can therefore be misleading.
Students should justify sample adequacy using the planned analysis, relevant methodological evidence, and realistic characteristics of the proposed instrument.
When development and substantive hypothesis testing use the same participants, additional concerns arise about overfitting, capitalization on chance, and the independence of validation evidence.
Can You Develop and Validate the Instrument Using the Same Sample?
Some development activities necessarily use the available sample to refine the instrument. However, evaluating a model on the same data used to select items and determine its structure may produce overly optimistic results.
For example, if researchers repeatedly remove items until a factor model fits one dataset well, the final structure may reflect sample-specific characteristics rather than a stable measurement model.
Independent validation samples or appropriately planned cross-validation procedures can provide stronger evidence of generalizability when feasible.
Nevertheless, splitting a small sample into separate development and validation groups may leave both groups inadequate for their intended analyses.
Researchers should evaluate the trade-off rather than automatically divide the sample. When independent validation is unavailable, they should report the preliminary nature of the findings and avoid claiming that the instrument has been comprehensively validated.
Can a Thesis Both Develop an Instrument and Test a Substantive Theory?
Yes, but the combined project may be substantially more demanding than either objective alone.
Instrument development requires evidence about the quality and interpretation of the measurements. Substantive theory testing requires evidence about the relationships or effects being investigated.
These objectives may involve different samples, analytical assumptions, and validation requirements.
For example, a dissertation that develops an AI literacy scale and then tests whether AI literacy predicts academic performance must address both the quality of the new scale and the adequacy of the predictive analysis.
Weaknesses in the measurement model may affect the substantive findings, even if the subsequent statistical analysis is technically correct.
Students should consider whether instrument development should constitute the primary research contribution rather than a preliminary activity added to an already ambitious study.
When Is Instrument Development a Suitable Main Dissertation Contribution?
Instrument development may be an appropriate central objective when the construct is important, existing measures are demonstrably inadequate, and the project can support a substantial program of conceptual and empirical evaluation.
For example, a doctoral project might focus on developing and evaluating a measure of a clearly defined educational technology construct for a specified population.
Such a project may contribute through conceptual clarification, item development, measurement structure, validity evidence, and an explicit account of limitations.
However, the degree's requirements must be considered. Some programs may accept a substantial instrument-development dissertation, while others expect a different form of theoretical or empirical contribution.
The student should establish the scholarly contribution and institutional expectations before committing to the design.
What If the Instrument Cannot Be Fully Validated Before Graduation?
Validation is an ongoing process of accumulating evidence for particular interpretations and uses. An initial study rarely establishes every possible measurement property across all populations and contexts.
Nevertheless, the evidence available at the time of use must be sufficient for the claims being made.
A preliminary instrument may be appropriate for an exploratory measurement study if its limitations are clearly identified. It may be unsuitable as the sole primary outcome in a definitive intervention evaluation.
Students should avoid presenting a newly developed questionnaire as fully validated merely because it underwent expert review, pilot administration, and one reliability analysis.
Watch Out
Do not use the phrase "validated questionnaire" as a blanket description when the available evidence supports only selected measurement properties. Specify what was evaluated, for which population, and which score interpretations remain uncertain.
When Should Students Avoid Developing a New Instrument?
New instrument development may be inadvisable when an appropriate established measure already exists, the student lacks necessary methodological support, the available sample cannot support the intended analyses, or the timeline cannot accommodate development and evaluation.
It may also be inappropriate when instrument development is not central to the research contribution and would divert resources from the substantive question.
In these circumstances, using an established instrument, conducting a justified adaptation, or narrowing the measurement objective may be preferable.
The decision should preserve measurement adequacy rather than simply reduce workload.
