03 · What You Need to Know
What Measurement Evidence Is Necessary Before the Main Study?
What Does It Mean to Establish Reliability and Validity Evidence?
Reliability concerns the consistency or precision of measurements under specified conditions. Validity concerns the extent to which evidence and theory support particular interpretations of scores for their intended uses.
These concepts are related but not interchangeable.
A questionnaire may produce highly consistent responses while measuring something different from the intended construct. Conversely, an instrument may contain relevant content but produce scores too imprecise for the intended comparisons.
The Standards for Educational and Psychological Testing emphasize that validity is associated with the interpretation and use of test scores rather than an unconditional property possessed by an instrument.
Consequently, researchers should avoid treating a questionnaire as permanently validated simply because an earlier study reported favorable measurement statistics.
What Is the Difference Between Reliability and Validity?
Reliability evidence
Addresses the consistency or precision of scores under specified measurement conditions, using methods appropriate to the instrument and intended interpretation.
Validity evidence
Supports the argument that scores can be interpreted and used as intended, drawing on relevant theoretical, content, response-process, structural, and other evidence.
For example, two assessors may consistently assign similar scores to students' research proposals. That agreement contributes reliability evidence.
However, if the rubric rewards formatting more heavily than research-design reasoning, consistent scoring does not establish that the assessment adequately represents research competence.
Both questions must be considered when the scores will support substantive research conclusions.
Why Is Measurement Evidence Particularly Important for New Instruments?
Established instruments may have accumulated evidence from previous applications, although that evidence must still be evaluated for relevance to the current population and purpose.
New instruments do not have the same measurement history.
Their item meanings, content coverage, dimensional structure, scoring rules, and precision may remain uncertain until systematically investigated.
For example, a newly developed questionnaire about AI literacy might combine knowledge, confidence, attitudes, and reported practices. Without a defensible measurement model, a total score may not represent a coherent attribute.
The decision to develop a new instrument rather than use an existing one therefore carries an obligation to establish appropriate supporting evidence.
What Types of Validity Evidence May Be Relevant?
The Standards identify several sources of validity evidence, including test content, response processes, internal structure, relationships with other variables, and consequences of testing.
Not every source is equally relevant to every instrument or intended use.
| Evidence Source |
Question Addressed |
Possible Evaluation |
| Test content |
Does the instrument adequately represent the intended construct? |
Construct blueprint, expert review, content analysis |
| Response processes |
Do respondents understand and complete the tasks as intended? |
Cognitive interviews, response-process observations |
| Internal structure |
Do relationships among items support the proposed scoring interpretation? |
Appropriate factor analysis or item response models |
| Relations with other variables |
Are score relationships consistent with defensible theoretical expectations? |
Convergent, discriminant, or criterion-related investigations |
| Consequences of testing |
What are the intended and unintended consequences of the proposed interpretation or use? |
Evaluation of decisions, impacts, and relevant consequences |
These sources contribute to an integrated validity argument. They should not be treated as independent certificates that can be checked off without considering the intended score interpretation.
Must Content Validity Be Examined Before Main Data Collection?
Content-related evidence is particularly important before administering a newly developed instrument because the items determine which aspects of the construct can be observed.
If essential dimensions are absent, collecting more responses will not automatically recover the missing content.
Researchers should therefore establish a clear construct definition, develop a content blueprint, and evaluate whether the proposed items adequately represent the intended domain.
Expert review and target-population involvement can contribute complementary evidence.
The COSMIN framework emphasizes relevance, comprehensiveness, and comprehensibility for patient-reported outcome measures. These considerations may inform other questionnaire-development contexts when applied appropriately.
Content review should be completed early enough that consequential problems can be corrected before the instrument is used as a primary research measure.
Does Expert Review Establish That a New Instrument Is Valid?
No. Experts may judge items conceptually relevant without knowing whether respondents interpret them correctly or whether the resulting scores support the proposed measurement model.
For example, experts may approve several statements about students' confidence in evaluating AI-generated information. Their approval does not establish that the items measure demonstrated evaluation competence.
Likewise, a favorable Content Validity Index summarizes particular expert judgments but does not establish reliability or structural validity.
The role of expert review before pilot testing is to improve and evaluate instrument content, not certify every measurement property.
What Role Does Cognitive Interviewing Play?
Cognitive interviewing examines how intended respondents understand questions, retrieve information, make judgments, and select responses.
It can identify misunderstandings that expert review or ordinary questionnaire completion may overlook.
For example, students may interpret "verifying AI-generated information" as rereading an AI response rather than checking claims against independent evidence.
If the intended construct concerns verification behavior, the difference could undermine item interpretation.
Conducting cognitive interviews during questionnaire development may allow researchers to address such problems before main data collection.
However, cognitive interviewing does not independently establish score precision or factor structure.
Which Reliability Evidence Should Be Established?
The appropriate form of reliability evidence depends on the instrument and the measurement conditions.
Internal consistency may be relevant when multiple items are intended to represent a common reflective construct.
Test-retest reliability may be relevant when the measured attribute is expected to remain sufficiently stable over the selected interval.
Inter-rater reliability or agreement may be necessary when scores depend on judgments made by observers or assessors.
For device-based measurements, repeatability and reproducibility may be more appropriate concerns.
Researchers should select reliability procedures according to the intended score and error sources rather than automatically calculating Cronbach's alpha for every instrument.
Is Cronbach's Alpha Enough to Establish Reliability?
No. Cronbach's alpha is an internal consistency coefficient whose interpretation depends on assumptions about the measurement model and item relationships.
It does not independently establish unidimensionality, temporal stability, inter-rater agreement, or validity.
A high alpha may arise partly from numerous highly correlated items, including items with redundant content.
Conversely, an instrument assessing several distinct components may not be appropriately evaluated through a single internal consistency coefficient.
Researchers should investigate dimensionality and select reliability estimates appropriate to the score and model.
The problem of redundant questionnaire items illustrates why internal consistency cannot substitute for adequate construct representation.
What Is Structural Validity, and When Should It Be Examined?
Structural validity concerns whether the relationships among observed items support the proposed dimensional structure of the instrument.
For example, a researcher may propose that a self-regulated learning questionnaire contains distinguishable planning, monitoring, and reflection dimensions.
Factor analysis may provide evidence about whether the observed item relationships are consistent with that model.
However, exploratory and confirmatory factor analyses address different objectives, and neither independently establishes the conceptual meaning of a factor.
When a new instrument's total or subscale scores depend on a particular dimensional structure, evidence about that structure may be especially important before those scores are used for confirmatory hypothesis testing.
For performance assessments, observational measures, and composite indices, different analytical approaches may be more appropriate.
Must Exploratory and Confirmatory Factor Analyses Be Completed Before the Main Study?
Not universally. The need depends on the instrument's measurement model, available evidence, and intended claims.
When dimensionality is uncertain, an exploratory analysis may help develop the proposed structure. Independent evaluation of the finalized model can provide stronger evidence than fitting and confirming it using the same observations.
However, splitting an inadequate sample into exploratory and confirmatory subsets does not automatically improve methodological quality.
Nor is factor analysis appropriate for every instrument. A performance rubric, formative index, or single-item measure may require different forms of evaluation.
Researchers should avoid treating the completion of particular statistical procedures as a universal definition of validation.
How Much Reliability Is Enough?
There is no universal coefficient that establishes sufficient reliability for every research purpose.
The acceptable degree of measurement error depends on whether scores are used for broad group comparisons, individual-level interpretation, screening, or high-consequence decisions.
For example, an instrument used to explore population-level patterns may tolerate different uncertainty than one used to make consequential judgments about individual students.
Reliability estimates should be accompanied by relevant uncertainty and interpreted in relation to the intended use.
Conventional cutoffs such as 0.70 or 0.80 may serve as preliminary reference points in some contexts, but they should not replace a justification grounded in measurement purpose.
Can Reliability Be Estimated From the Main Study Data?
Yes. Reliability evidence is often evaluated in the sample and conditions of actual use, and reporting it can be informative.
However, estimating reliability after main data collection does not resolve all risks of using an insufficiently evaluated instrument.
If the measure lacks essential content, contains misunderstood questions, or has an unsupported scoring structure, a favorable reliability coefficient cannot correct those problems.
Furthermore, revising the instrument or selecting items after examining the main results may introduce data-dependent decisions that complicate substantive inference.
Researchers should therefore distinguish routine evaluation of measurement performance in the main sample from foundational development that ideally should occur before strong confirmatory conclusions depend on the scores.
Can Validity Evidence Be Established Using Main Study Data?
Some validity evidence can be obtained from main study data, provided the design supports the intended investigation.
For example, researchers may evaluate a prespecified measurement model or examine theoretically expected relationships with other variables.
However, using the same data to select items, revise a model, and then claim independent confirmation creates a risk of capitalization on chance.
The concern becomes particularly serious when items are selected because they produce favorable relationships with the study's primary outcome.
Such decisions can make the substantive association partly a consequence of instrument construction.
When development is substantial, conducting instrument development in a separate study or phase may provide stronger evidence.
What Is the Difference Between Pilot Testing and Validation?
Pilot testing commonly examines feasibility and practical functioning, such as administration time, respondent comprehension, data recording, and procedural problems.
A measurement-properties study may examine reliability, internal structure, measurement error, or relationships with external variables.
These activities can overlap, but a pilot is not automatically a comprehensive validation study.
Pilot testing
Investigates whether the instrument and associated procedures can function as intended, potentially identifying problems requiring revision.
Measurement evaluation
Investigates whether evidence supports the intended score interpretations, including relevant measurement properties and assumptions.
A pilot involving a small convenience sample may identify confusing wording but provide insufficient precision for evaluating a complex factor structure.
Researchers should report what the pilot actually established rather than relying on the label alone.
Does Every New Instrument Need a Separate Validation Study?
No universal requirement mandates a separate study or publication.
Development and evaluation may occur within a coordinated multiphase project, provided the stages are appropriately designed and their evidentiary roles are distinguished.
For example, a researcher might conduct qualitative item development, expert review, cognitive interviews, and preliminary measurement evaluation before administering a finalized instrument in an intervention study.
Whether those activities are published separately is an editorial rather than inherently methodological decision.
The important question is whether the evidence available before substantive inference is adequate for the intended use.
What If the Instrument Is Intended Only for Exploratory Research?
Exploratory research may legitimately use provisional instruments when developing or evaluating the measurement procedure is part of the research objective.
However, the limitations should be stated explicitly.
For example, researchers may investigate how students interpret an emerging construct and explore preliminary item relationships without claiming that the resulting scores provide a definitive measure.
Substantive conclusions should be framed as provisional when measurement uncertainty is substantial.
The distinction is not between instruments that are permitted and prohibited, but between claims that the available evidence can and cannot support.
What If the Main Study Is an Intervention Trial?
Measurement uncertainty can be especially consequential when a new instrument serves as the primary outcome of an intervention study.
If item wording, scoring rules, or dimensional structure are revised after observing intervention effects, the estimated effect may be influenced by those decisions.
Researchers should therefore aim to establish a stable instrument and prespecified scoring procedure before evaluating the intervention.
Appropriate baseline and follow-up comparability, sensitivity to relevant change, and measurement precision may also require investigation.
When the instrument is not sufficiently developed, a preliminary measurement study or revised outcome strategy may be preferable to relying on uncertain scores for strong causal claims.
Should Measurement Invariance Be Evaluated Before Group Comparisons?
When the research intends to compare scores across groups, evidence that the instrument functions sufficiently comparably may be necessary.
For example, a questionnaire may be administered to students from different language groups or educational contexts.
Identical wording or a common factor structure does not automatically establish equivalent score interpretation.
Measurement invariance or differential item functioning analyses may be informative when appropriate to the instrument and sample.
The level of evidence needed depends on the comparison, measurement model, and intended inference.
Researchers should consider differential measurement error between study groups rather than assuming that the same instrument necessarily operates identically across populations.
What Should Be Done If a New Instrument Performs Poorly During Evaluation?
The appropriate response depends on the nature and severity of the problem.
If cognitive interviews reveal ambiguous wording, revision and further testing may resolve the difficulty.
If the proposed dimensional structure is unsupported, researchers may need to reconsider the measurement model, item content, or construct definition.
If scores are too imprecise for the intended comparison, additional items, different scoring procedures, or an alternative instrument may be necessary.
Researchers should avoid automatically deleting items until a desired coefficient or model-fit threshold is reached.
Any consequential revision should be documented and evaluated for its effects on construct coverage and score interpretation.
How Should Reliability and Validity Evidence Be Reported?
Reports should identify the instrument's development status, intended construct, target population, scoring procedure, and available supporting evidence.
Describe the studies and samples used to evaluate content, response processes, internal structure, and relevant reliability properties.
Where estimates are reported, provide appropriate uncertainty and explain their interpretation.
Distinguish evidence obtained before the main study from analyses conducted using the main dataset.
Avoid statements such as "the instrument was validated because Cronbach's alpha exceeded 0.70."
A stronger report explains which interpretations the evidence supports and which limitations remain.
Watch Out
Do not treat expert approval, pilot completion, Cronbach's alpha, or favorable factor-model fit as a standalone certificate of validity. These procedures provide different forms of evidence, and none automatically establishes that a new instrument is appropriate for every population or research purpose.
