03 · What You Need to Know
How to Separate Instrument Development From Substantive Research
What Is the Difference Between an Instrument-Development Study and a Main Research Study?
An instrument-development study investigates how a construct should be operationalized and whether the resulting observations support the intended score interpretations.
Its questions may concern item relevance, comprehensibility, dimensional structure, reliability, scoring, and relationships with other variables.
A substantive study uses measurements to investigate a different research question, such as whether an intervention improves learning, whether motivation predicts persistence, or whether a construct differs between populations.
These activities are connected because substantive conclusions depend on measurement quality. However, evidence that an instrument functions appropriately is not the same as evidence supporting a substantive hypothesis.
Instrument-development research
Establishes the conceptual, procedural, and empirical basis for interpreting measurements obtained through an instrument.
Substantive research
Uses measurements to investigate theoretical relationships, group differences, intervention effects, or other phenomena of interest.
A project may legitimately contain both activities. The important issue is whether their different objectives and evidentiary requirements are recognized in the design.
Why Might a Separate Instrument-Development Study Be Preferable?
Developing a new instrument often requires several rounds of decisions before a stable version is available.
Researchers may revise item wording, remove poorly functioning items, reconsider dimensional structure, or change scoring procedures. Such decisions are expected during development.
However, they create complications when the same dataset is subsequently treated as an independent test of the final instrument or the substantive hypotheses.
A separate study can provide time and analytical independence for investigating the measurement model before the instrument is used for its primary research purpose.
It may also allow researchers to recruit participants specifically suited to development, conduct cognitive interviews, examine measurement properties, and revise the instrument without disrupting a planned intervention or confirmatory analysis.
These advantages are especially relevant when developing a genuinely new research instrument requires substantial conceptual and empirical work.
Does Separate Study Mean Separate Data Collection?
Not necessarily, although independent data collection may be valuable.
A separate development study can use a distinct sample collected specifically for measurement evaluation. A larger project may also contain separate development and evaluation phases within one coordinated research protocol.
In some circumstances, researchers can divide a sufficiently large dataset into development and evaluation subsets. However, the adequacy of this approach depends on sample size, population coverage, and the extent to which the evaluation data remain independent of instrument-selection decisions.
Splitting a small dataset may leave both subsets insufficient for the intended analyses.
Furthermore, a random split of one sample evaluates performance in another subset of that sampling context. It does not necessarily establish that the instrument generalizes to a different institution, language group, or population.
The distinction is therefore between analytical independence, population generalizability, and administrative separation. They are related but not interchangeable.
Does Instrument Development Have to Be Published Separately?
No. A separate publication is an editorial decision rather than a universal methodological requirement.
A researcher may report development and subsequent substantive analysis in one sufficiently detailed manuscript, particularly when both are integral to the research contribution.
Alternatively, an instrument-development article may provide a fuller account of construct definition, item generation, response processes, and psychometric evidence, followed by later studies applying the instrument.
The appropriate approach depends on the scope of the work, journal expectations, and whether each manuscript makes a distinct scholarly contribution.
Publishing two papers does not itself establish methodological independence. If both analyses rely on the same data, that dependence must still be reported.
Similarly, presenting two genuinely separate phases in one article does not invalidate their methodological distinction.
What Are the Main Design Options?
| Design |
Potential Advantage |
Important Limitation |
| Separate development study followed by main study |
Allows substantial development and evaluation before substantive testing |
Requires additional time, recruitment, and resources |
| Multiphase project with separate samples |
Integrates development and substantive research within one coordinated project |
Requires careful planning of phase transitions and sufficient participants |
| Development and evaluation subsets within one dataset |
May support internal cross-validation without another recruitment period |
Can reduce analytical sample sizes and provide limited external generalizability |
| Existing instrument with supplementary measurement evaluation |
May avoid unnecessary development when the instrument is already conceptually suitable |
Population-specific evidence may still be required |
| Exploratory development embedded in the main study |
May be feasible when the project is explicitly exploratory |
Limits the strength of confirmatory claims based on the same data |
These designs should be selected according to the measurement question and intended conclusions. A separate study is not automatically superior if it is poorly designed or evaluates an irrelevant population.
What Is the Problem With Developing and Confirming a Scale Using the Same Data?
Instrument development often involves data-dependent decisions.
Researchers may remove items with weak correlations, choose a factor solution that fits the sample, or revise the scoring model after inspecting results.
When the resulting model is evaluated using the same observations, fit statistics and other estimates may be overly optimistic because the model was selected partly to fit those data.
This is a form of overfitting or capitalization on chance.
For example, a factor structure identified through exploratory factor analysis should not be described as independently confirmed merely because a confirmatory factor analysis of the same data produces acceptable fit.
The confirmatory analysis may still provide descriptive information about the fitted model, but it does not constitute an independent test of the development decisions.
Independent evaluation is particularly valuable when the instrument will be used to support strong claims about relationships or intervention effects.
Should Exploratory and Confirmatory Factor Analyses Use Different Samples?
Using independent samples for exploratory and confirmatory analyses is often methodologically advantageous when the purpose is to develop and then test a proposed factor structure.
Exploratory factor analysis can investigate plausible dimensional structures, while confirmatory factor analysis evaluates a specified model under its assumptions.
When both analyses use the same observations after data-driven model selection, the confirmatory stage does not provide independent evidence of generalizability.
However, separate samples are not a universal requirement for every measurement investigation. The appropriate design depends on whether the model was prespecified, the analytical objectives, sample availability, and the strength of the intended claims.
Researchers should also avoid treating factor analysis as the complete validation process. Content validity, response processes, reliability, and other evidence may remain necessary.
Can You Split One Sample Into Development and Validation Groups?
Yes, when the available sample is sufficiently large and the split supports the planned analyses.
For example, researchers may use one subset to explore item structure and another to evaluate the prespecified model.
The evaluation subset should remain protected from the decisions made during development. Repeatedly inspecting and modifying the model using both subsets undermines the intended independence.
However, splitting a dataset does not automatically provide adequate sample sizes for factor analysis or other measurement evaluations.
Sample-size planning should consider the number of factors and indicators, item distributions, communalities, model complexity, and intended precision rather than relying solely on a fixed respondent-to-item ratio.
When the total sample is limited, a carefully reported exploratory analysis may be more defensible than two underpowered analyses presented as separate validation stages.
When Is a Separate Development Study Especially Important?
Separate development becomes particularly attractive when the proposed instrument has an uncertain conceptual structure, requires extensive item refinement, or will provide the primary outcome for an important substantive investigation.
It may also be valuable when researchers need to evaluate a scoring rubric, establish inter-rater reliability, calibrate assessment tasks, or investigate whether the instrument functions appropriately across populations.
For example, an intervention trial using a newly created performance assessment may be difficult to interpret if the assessment's scoring rules are still being revised after intervention data are collected.
A preliminary development phase can help establish a stable measurement procedure before the trial begins.
The need is not necessarily for a separate journal article, but for sufficient measurement evidence and procedural stability.
When Can Instrument Development Be Embedded Within the Main Project?
Integrated development may be appropriate when the project explicitly includes measurement development as a research objective and uses a design that distinguishes development from subsequent evaluation.
For example, a multiphase study may begin with qualitative construct exploration, proceed to item development and pilot testing, and then administer a finalized instrument in a later substantive phase.
These activities may be conducted under one research protocol while retaining distinct analytical purposes.
Another possibility is an exploratory study in which a provisional instrument is used to investigate an emerging construct. In that case, the researcher should characterize the measurement evidence as preliminary and limit substantive claims accordingly.
What is problematic is not combining activities within one project, but treating an instrument developed and optimized on the main analytical data as though its performance had already been independently established.
How Should the Development and Main Study Samples Be Related?
Development participants should provide evidence relevant to the intended population and measurement purpose.
For example, an instrument intended for university students should not be evaluated exclusively among research experts or a convenience sample of unrelated professionals.
However, independent evaluation among participants from another relevant institution or context may help investigate generalizability.
Differences between development and main-study populations should be examined because the instrument may function differently across groups.
The researcher should consider population and contextual appropriateness rather than assuming that evidence from any sample transfers automatically.
What Evidence Should Be Established Before the Main Study?
The evidence needed before substantive use depends on the instrument and intended interpretation.
For a newly developed questionnaire, relevant evidence may include content validity, comprehensibility, response processes, structural validity, and reliability.
For a performance assessment, task representativeness, scoring consistency, and relationships with relevant external indicators may be especially important.
When the main study compares groups or evaluates change, measurement invariance, measurement error, or responsiveness may also require attention.
The COSMIN Study Design Checklist illustrates how different measurement properties require different study designs and analytical procedures. Its specific framework concerns patient-reported outcome measures, so its application to other instruments should be appropriately adapted. COSMIN's official methodological resources distinguish these requirements.
Researchers should avoid claiming that an instrument has been comprehensively validated because it passed expert review, a small pilot, or one internal consistency analysis.
Does Every Instrument Require a Completely Separate Validation Study?
No. The need for additional evaluation depends on the existing evidence and the changes introduced.
An established instrument administered without modification in a closely comparable population may already have substantial relevant evidence.
A translated, shortened, or otherwise adapted version may require additional evaluation, but the necessary scope depends on how the modification affects the intended score interpretation.
Newly developed instruments generally require more foundational evidence because their content, structure, and scoring have not yet accumulated an established research history.
Researchers should distinguish the absence of evidence from evidence of poor measurement. These situations may require different responses.
How Can the Same Dataset Create Circularity in Substantive Hypothesis Testing?
Circularity can arise when researchers select instrument items because they correlate strongly with the outcome they later claim the instrument predicts.
For example, suppose a researcher develops a learning motivation scale and retains only items that show strong associations with academic achievement in the development dataset.
Using the resulting scale to demonstrate a strong relationship with achievement in that same dataset does not provide an independent test of the substantive hypothesis.
The association has partly influenced the construction of the measure itself.
Independent evaluation or carefully planned validation procedures can help distinguish genuine predictive relationships from those amplified by outcome-guided item selection.
Researchers should document whether external variables were used during item selection and how that decision affects subsequent inference.
Can Statistical Cross-Validation Replace an Independent Study?
Cross-validation can provide useful information about whether a measurement model or prediction procedure performs similarly in data not used for its initial fitting.
However, different cross-validation approaches address different questions.
Randomly partitioning observations from one population may assess internal generalizability, but it does not necessarily establish that the instrument functions similarly in another language, institution, or cultural context.
Cross-validation also requires care when observations are clustered, repeated, or otherwise dependent. Splitting individual records without accounting for those dependencies can create information leakage.
Independent samples collected under relevant conditions may provide stronger evidence about external applicability, although they are not automatically free from measurement problems.
What If You Cannot Afford a Separate Development Study?
Resource constraints are legitimate considerations, but they do not remove the need for defensible measurement.
Researchers may consider a staged project, collaboration with another research team, a suitable existing instrument, or a narrower exploratory objective.
Where one dataset must serve several purposes, the development and evaluation decisions should be planned carefully, and the limitations of nonindependent analyses should be reported.
It may be preferable to produce a well-documented preliminary instrument rather than make strong confirmatory claims from a scale whose measurement properties remain uncertain.
The relevant choice should follow the trade-offs between measurement quality and available resources.
How Should an Integrated Development and Main Study Be Reported?
Reports should distinguish the objectives, participants, procedures, and analyses associated with instrument development from those associated with substantive testing.
Describe when items were generated, revised, removed, and finalized. Identify which datasets informed those decisions and which were reserved for subsequent evaluation.
Explain the evidence supporting the intended score interpretations and acknowledge where independent confirmation remains unavailable.
If the same participants contributed to several phases, report that dependence explicitly.
Transparent reporting is more informative than simply labeling an early activity as a validation phase without specifying what evidence it produced.
Watch Out
Do not describe a measurement model as independently confirmed when its structure was selected, revised, and evaluated using the same observations without an independent test. Likewise, do not select items based on their relationship with the main outcome and then treat that relationship as independent evidence supporting the study hypothesis.
