03 · What You Need to Know
How Expert Review Contributes to Research Instrument Development
What Is Expert Review in Instrument Development?
Expert review is a structured evaluation of proposed instrument content by individuals whose knowledge is relevant to the construct, target population, measurement method, or intended use.
Reviewers may examine whether items represent the intended construct, whether important dimensions are missing, whether wording is technically accurate, and whether response options or scoring criteria are appropriate.
For example, a questionnaire measuring perceived academic self-efficacy may require reviewers familiar with self-efficacy theory and educational measurement. A performance assessment of clinical reasoning may additionally require specialists familiar with the tasks and professional standards involved.
The purpose is not simply to obtain approval from respected academics. It is to generate documented judgments that can improve the content and interpretation of the instrument.
Expert review should therefore begin with a clear construct definition and content blueprint, as explained in defining constructs before developing instrument items.
Why Should Expert Review Usually Occur Before Pilot Testing?
Pilot testing is more informative when the instrument has already undergone a reasonable conceptual and technical review.
Experts may identify items that are irrelevant, misleading, theoretically inconsistent, or insufficiently representative of the construct. Addressing these problems before pilot administration can reduce unnecessary participant burden and prevent researchers from collecting responses to items that require substantial revision.
For instance, an item intended to measure actual research competence might ask participants how confident they feel about conducting research. An expert could identify that the item concerns perceived competence rather than demonstrated ability.
Pilot participants may answer the question without difficulty, but their ability to respond would not resolve the conceptual mismatch.
Expert review and pilot testing therefore serve complementary purposes. The former can evaluate substantive content before administration, while the latter can investigate how the proposed instrument operates in practice.
What Should Experts Evaluate?
The review criteria should follow the intended instrument and measurement purpose.
| Review Criterion |
Question for Reviewers |
Potential Finding |
| Construct relevance |
Does this item represent the specified construct or dimension? |
An item measures motivation rather than self-efficacy |
| Content coverage |
Are essential aspects of the construct represented? |
An important dimension is missing |
| Conceptual accuracy |
Does the wording reflect the intended theoretical meaning? |
A technical concept is described incorrectly |
| Clarity |
Is the wording sufficiently precise for its intended purpose? |
A statement contains two separate judgments |
| Response suitability |
Do the response options match the judgment requested? |
Agreement categories are used for a frequency question |
| Scoring appropriateness |
Can responses be scored consistently and meaningfully? |
A rubric does not distinguish performance levels adequately |
| Population relevance |
Does the content assume experiences unavailable to intended respondents? |
An item assumes access to laboratory facilities |
Not all criteria should necessarily be collapsed into a single rating. An item may be relevant but unclear, or clearly written but conceptually inappropriate.
Separating evaluation criteria helps researchers identify the nature of the problem and select an appropriate revision.
Who Qualifies as an Expert Reviewer?
Expertise should be defined in relation to the reviewer's assigned task rather than academic rank alone.
Relevant qualifications may include theoretical scholarship, instrument-development experience, professional practice, familiarity with the target population, or expertise in the measurement method.
For example, an instrument concerning educational technology may benefit from reviewers with knowledge of learning theory and technology-mediated instruction, alongside individuals experienced in educational measurement.
A researcher who understands the subject matter may not necessarily have expertise in psychometric evaluation. Similarly, a measurement specialist may not know the disciplinary content well enough to judge whether important aspects have been omitted.
Review panels can therefore benefit from complementary expertise.
The research team should document why each reviewer was selected and what their expertise contributes.
How Many Experts Are Needed?
There is no universally sufficient number of experts for every instrument-development study.
The appropriate panel size depends on the breadth and complexity of the construct, the expertise required, the review method, and whether quantitative agreement indices will be calculated.
Some content-validity procedures use relatively small expert panels, but the interpretation of their agreement statistics depends heavily on the number of reviewers.
For example, agreement among three experts is numerically sensitive to a single dissenting rating. Larger panels may provide broader perspectives but do not automatically improve the quality of judgment.
Researchers should avoid claiming that a particular number of experts is universally mandatory or sufficient.
Panel composition, independence, expertise, and review procedures deserve at least as much attention as panel size.
Should All Experts Have the Same Background?
Not necessarily. A multidisciplinary panel may identify problems that a homogeneous panel would overlook.
Subject specialists can evaluate conceptual and technical accuracy. Measurement specialists can examine item construction, scoring, and response formats. Practitioners familiar with the target setting may identify contextual assumptions that academic reviewers miss.
However, reviewers should not be asked to judge areas outside their competence without qualification.
For example, a psychometrician may appropriately evaluate whether a questionnaire item is double-barreled without claiming expertise in the disciplinary accuracy of its technical content.
Researchers may therefore assign different review criteria to different specialists while retaining a coherent overall review process.
What Materials Should Experts Receive?
Experts need more than an isolated list of questionnaire statements.
A structured review package should ordinarily include the construct definition, intended population, measurement purpose, proposed dimensions, item wording, response options, and instructions for evaluation.
Where relevant, reviewers should also receive the content blueprint, scoring rubric, and explanation of how the instrument will be administered.
Without this information, reviewers may apply different personal definitions of the construct and produce judgments that are difficult to reconcile.
For example, an item may seem irrelevant to a reviewer who assumes the instrument measures general digital competence but appropriate to another who understands its narrower focus on perceived information-evaluation confidence.
Providing a common conceptual specification helps make disagreements more interpretable.
Should Expert Review Be Qualitative, Quantitative, or Both?
Expert review may use narrative feedback, structured ratings, or a combination of approaches.
Qualitative comments can explain why an item is problematic and suggest revisions. Quantitative ratings can summarize patterns of judgment across reviewers.
For example, reviewers might rate each item's relevance using a four-category scale and provide written comments about ambiguity, redundancy, or missing content.
Neither approach is automatically superior. Numerical agreement is useful only when the rating question is meaningful and the reviewers understand the criteria consistently.
A combination of ratings and explanatory comments can be particularly informative because agreement indices alone do not explain the reasons for disagreement.
What Is the Content Validity Index?
The Content Validity Index, commonly abbreviated CVI, summarizes expert ratings of item or scale content under a specified rating procedure.
A commonly used item-level index, I-CVI, is calculated as the proportion of experts rating an item as sufficiently relevant, often using ratings of 3 or 4 on a four-category relevance scale.
The index summarizes the proportion of favorable judgments. It does not establish that the item measures the construct accurately or that respondents interpret it correctly.
Researchers should specify the rating scale, the relevance threshold, and how missing or incomplete expert ratings were handled.
How Should I-CVI Values Be Interpreted?
Interpretation depends on panel size, the rating procedure, and the purpose of the review.
Lynn's influential work on content validity and the subsequent discussion by Polit, Beck, and Owen provide commonly cited approaches to evaluating expert agreement.
For example, Polit and colleagues discuss an I-CVI of approximately 0.78 or higher as a useful criterion for panels with six or more experts under their proposed framework.
That recommendation is not a universal rule establishing item validity. Its appropriateness depends on the review design and the assumptions underlying the index.
Researchers should also consider whether an item addresses essential content that would otherwise be omitted. A low agreement value may warrant investigation rather than automatic deletion.
What Is the Scale-Level Content Validity Index?
A scale-level CVI summarizes content judgments across multiple items.
Two commonly discussed approaches are S-CVI/Ave, the average of item-level CVIs, and S-CVI/UA, the proportion of items receiving universal agreement among reviewers.
These indices answer different questions and can produce substantially different values.
For example, an instrument may have high average item relevance while relatively few items receive unanimous approval.
Researchers should specify which calculation was used rather than reporting a single unexplained "CVI."
A high scale-level index does not guarantee that every important dimension is represented. Several highly rated items can still omit a central aspect of the construct.
What Is the Content Validity Ratio?
The Content Validity Ratio, or CVR, was proposed by Lawshe as a way to summarize judgments about whether an item is essential.
It differs from a relevance-based CVI because it asks reviewers to classify essentiality rather than merely relevance.
Interpretation requires attention to panel size and the relevant critical-value approach. The same numerical CVR does not have identical evidentiary implications for every panel size.
CVR and CVI should not be treated as interchangeable or as definitive measures of instrument validity.
Does High Expert Agreement Establish Content Validity?
Expert agreement contributes evidence about content relevance, but content validity requires a broader argument.
For example, all reviewers might agree that a set of items measures students' confidence in using AI tools. That agreement would not establish that the items adequately measure demonstrated AI evaluation competence.
Similarly, reviewers may rate each existing item highly while failing to identify a missing construct dimension.
The COSMIN methodology emphasizes relevance, comprehensiveness, and comprehensibility when evaluating content validity for patient-reported outcome measures. These considerations can inform other questionnaire contexts when applied appropriately.
Content-related evidence should therefore consider the entire instrument and intended interpretation, not merely favorable ratings of individual statements.
What Should You Do When Experts Disagree?
Disagreement may identify ambiguity, competing theoretical assumptions, or differences in professional experience.
Researchers should examine the reasons behind conflicting judgments rather than simply following the majority.
For example, one expert may reject an item because it measures an outcome rather than the construct itself, while another may support it because the behavior is commonly associated with the construct.
The disagreement may reveal a conceptual issue requiring clarification before item selection.
Researchers can revise the item, seek additional explanation, conduct another review round, or retain it provisionally for further investigation.
Decisions should follow the construct definition and available evidence, not the academic seniority of the most influential reviewer.
Should Every Expert Suggestion Be Implemented?
No. Expert comments are evidence for consideration rather than instructions that must all be adopted.
Some suggestions may conflict with the construct definition, introduce double-barreled wording, or unnecessarily increase instrument length.
Other suggestions may be valuable but concern a different research purpose.
The research team should document whether each consequential recommendation was accepted, modified, or declined, together with the rationale.
When experts disagree, the team should consider whether additional theoretical or target-population evidence is needed.
Mechanical acceptance of every suggestion can produce a less coherent instrument than a carefully justified selection process.
Should Experts Review Individual Items or the Entire Instrument?
Both levels matter.
Item-level review can identify unclear wording, poor relevance, or unsuitable response formats.
Instrument-level review considers whether the item set adequately represents the construct, whether dimensions are balanced, and whether the scoring approach supports the intended interpretation.
For example, a questionnaire may contain individually appropriate items but devote nearly all its content to one dimension while neglecting another essential component.
Reviewing only isolated items could miss this imbalance.
Experts should therefore have opportunities to comment on the overall content framework and identify missing material.
How Is Expert Review Different From Target-Population Review?
Experts and intended respondents contribute different forms of evidence.
Experts may be particularly well positioned to judge conceptual relevance, technical accuracy, and alignment with a theoretical framework.
Target-population members can reveal whether questions reflect their experiences and whether they understand the wording as intended.
Neither perspective fully replaces the other.
For example, experts may approve a technically accurate statement that students interpret inconsistently. Conversely, students may find an item easy to answer even though it does not represent the intended construct.
The importance of target-population review of instrument items follows from this distinction.
Can Expert Review Replace Cognitive Interviewing?
No. Expert review and cognitive interviewing investigate different aspects of the measurement process.
Cognitive interviewing examines how intended respondents understand questions, retrieve information, form judgments, and select responses.
Experts may predict possible misunderstandings, but their own comprehension does not establish how participants will interpret the instrument.
For questionnaires involving unfamiliar terminology, complex recall, or context-sensitive judgments, cognitive interviewing may reveal problems that expert ratings do not identify.
Researchers should consider cognitive interviewing during questionnaire development when response-process evidence is consequential.
Should Experts Review Revised Items Again?
A second review may be appropriate when revisions substantially change item meaning, construct coverage, response options, or scoring.
Minor grammatical corrections may not require another full panel review, although the decision depends on the instrument and intended use.
For example, changing a statement from perceived confidence to demonstrated ability would be a substantive modification requiring renewed conceptual evaluation.
Researchers should distinguish editorial changes from changes that alter the measurement task.
Repeated review should have a clear purpose rather than continue indefinitely until every reviewer expresses complete agreement.
How Should Expert Review Be Reported?
Reports should describe the reviewers' relevant expertise, selection rationale, review criteria, rating procedures, and methods used to interpret feedback.
If quantitative indices are reported, identify the formulas, rating thresholds, number of reviewers, and treatment of incomplete ratings.
Explain how the review influenced item retention, revision, deletion, or addition.
Where substantial disagreement occurred, report how the research team addressed it.
A statement such as "the questionnaire was validated by five experts" is insufficient because it does not identify what was evaluated or what evidence was obtained.
More defensible reporting describes the review as a source of content-related validity evidence within a broader instrument-development process.
Watch Out
Do not describe a newly developed instrument as fully validated because experts approved its items or because its Content Validity Index exceeded a conventional threshold. Expert review contributes content-related evidence, but response processes, reliability, internal structure, and other measurement properties may still require investigation.
