Can Generative AI Replace Expert Content-Validity Review of a Questionnaire?

Can ChatGPT Serve as an Expert Validator of Your Questionnaire?

You have developed a questionnaire and need to establish whether its items adequately represent the construct you intend to measure. Traditionally, you might invite qualified researchers or subject-matter specialists to review the instrument.

But generative AI can perform what appears to be a similar task. You can provide a construct definition, submit your questionnaire items, and ask ChatGPT to rate each statement for relevance, clarity, and representativeness. It may return a detailed evaluation table, recommendations for revision, and numerical ratings resembling those from an expert panel.

If AI can produce these assessments almost immediately, is it still necessary to involve human experts? More importantly, can AI-generated ratings legitimately be reported as evidence of content validity?

No, AI Feedback Is Not Equivalent to Independent Expert Review

What Expert Content-Validity Review Provides That AI Cannot Simply Reproduce

What Is Expert Content-Validity Review?

Expert content-validity review is a systematic examination of whether an instrument's content adequately represents the intended construct or domain. Reviewers evaluate the relationship between the items, the construct definition, the intended respondents, and the proposed measurement purpose.

For example, suppose a researcher develops an academic self-efficacy scale for undergraduate students. Reviewers might examine whether its items represent perceived capability to perform academic tasks rather than related concepts such as motivation, enjoyment, or previous academic achievement.

They may also identify important dimensions that have been omitted, question whether certain items are appropriate for the target population, and recommend changes to the instrument.

The COSMIN framework describes content validity in terms of relevance, comprehensiveness, and comprehensibility. Its methodology emphasizes evaluating the content in relation to the construct, population, and context of use. The COSMIN content-validity methodology provides a detailed account of these considerations.

Although COSMIN was developed primarily for health-related outcome measurement, its conceptual distinctions are informative for educational and social research. Its specific procedural standards should not automatically be treated as universal requirements for every questionnaire.

Content Validity Involves More Than Rating Individual Items

Researchers sometimes reduce expert validation to asking several reviewers whether each item is relevant. Item relevance is important, but it does not fully address whether the questionnaire adequately represents the intended content domain.

Qualified experts may be particularly valuable when evaluating theoretical relevance and content coverage. Intended respondents are especially important when evaluating comprehensibility and the response process.

Consequently, even a well-conducted expert review should not automatically be considered a substitute for cognitive interviewing or respondent pretesting.

Why Are AI-Generated Judgments Different From Human Expert Judgments?

When an AI model evaluates questionnaire items, it generates an assessment based on the supplied information and patterns learned during training. It may produce useful explanations, recognize familiar methodological problems, and identify inconsistencies with a construct definition.

However, the model's output does not establish that an independently recruited subject-matter expert has evaluated the instrument.

A documented expert panel offers identifiable qualifications, relevant disciplinary experience, a defined review procedure, and judgments that can be examined in relation to the reviewers' expertise.

Experts may disagree, and their assessments can be affected by their own assumptions or theoretical preferences. Nevertheless, their judgments arise from an accountable review process rather than from a model generating a plausible evaluation.

The distinction is not that human reviewers are always correct and AI is always incorrect. Rather, the two activities produce different kinds of information, with different evidential and accountability characteristics.

Can AI Act as Several Independent Expert Reviewers?

A researcher might ask ChatGPT to evaluate the same questionnaire five times, assigning a different expert identity to each response. The resulting ratings may appear to represent a panel of five reviewers.

However, assigning different roles or personas to one model does not create five independent qualified experts.

Even when separate AI systems are used, their outputs should not automatically be treated as independent human expert judgments. They may share training sources, exhibit similar biases, or respond to the same framing assumptions.

AI-generated reviewer simulations may be useful for anticipating potential criticisms or comparing alternative interpretations. They should be identified as simulated assessments, not reported as an actual expert validation panel.

Can AI Calculate the Content Validity Index?

Yes, when supplied with genuine reviewer ratings and an appropriate calculation procedure.

The Content Validity Index, or CVI, is commonly used to summarize expert judgments about item relevance. One widely used item-level form, the I-CVI, is calculated as the proportion of reviewers who rate an item as relevant under the selected rating procedure.

Polit, Beck, and Owen (2007) discuss the usefulness and limitations of CVI procedures, including the problem of chance agreement. They suggest that an I-CVI of at least 0.78 with three or more experts may provide favorable evidence under their proposed approach. This is a methodological recommendation, not a universal threshold that independently establishes validity.

AI can calculate the index from authentic ratings, check arithmetic, and help organize reviewer comments. It cannot create genuine expert agreement by generating its own ratings.

What About AI-Generated Relevance Scores?

Suppose AI rates each item from 1 to 4 according to how relevant it appears to the supplied construct definition. Such ratings may help prioritize items for further examination.

However, these scores represent the model's output under a particular prompt and configuration. They should not be described as ratings obtained from qualified independent experts.

AI-generated scores may also change when the construct definition is phrased differently, when additional context is provided, or when a different model is used.

A high score therefore does not independently establish content validity. Researchers should examine the reasoning behind any flagged item and evaluate it against the theoretical framework.

Can AI Identify Content That Expert Reviewers Might Miss?

Potentially. AI can examine a large item pool systematically and compare the wording of each statement with a supplied construct definition. It may flag repeated content, inconsistencies, or obvious conceptual mismatches that deserve further attention.

For example, an AI system reviewing a scale about research self-efficacy might identify that several items concern research motivation rather than perceived capability.

Such observations can improve the preparation of an instrument before expert review.

Nevertheless, AI may also overlook important content gaps, misunderstand theoretical distinctions, or suggest revisions that change the construct. Its apparent thoroughness should not be confused with demonstrated accuracy.

Researchers should therefore treat AI feedback as a supplementary source of candidate concerns rather than as a definitive evaluation.

How Should You Prompt AI for a Preliminary Content Review?

The prompt should provide the construct definition, proposed dimensions, target population, and candidate items. It should also explicitly prohibit unsupported validation claims.

The resulting assessment can be used to refine the draft or prepare questions for expert reviewers.

However, AI should not be allowed to redefine the construct without theoretical justification. A model can produce a convincing explanation for a conceptually inappropriate item.

How Can AI Support a Genuine Expert Review Process?

AI can assist with the administrative and analytical activities surrounding expert review without replacing the reviewers themselves.

For example, researchers may use AI to prepare a review form, organize item-to-construct mappings, summarize de-identified qualitative feedback, or generate a revision log.

When experts provide ratings, AI-assisted calculations may help produce item-level and scale-level summaries. The researcher must verify the calculations and retain the original reviewer data.

AI may also help identify disagreements that require further discussion. However, summarizing disagreement is different from deciding which expert judgment should prevail.

The final decisions should be justified in relation to the construct definition, reviewer evidence, respondent needs, and the intended use of the instrument.

Does Expert Review Alone Establish Questionnaire Validity?

No. Expert content review contributes evidence concerning the adequacy of the instrument's content. It does not independently establish every aspect of validity.

For a multi-item psychometric scale, researchers may additionally need evidence concerning respondent interpretation, internal structure, reliability, and relationships with other variables.

Even unanimous expert agreement does not guarantee that respondents will interpret the items as intended or that the resulting scores will support the proposed measurement model.

Likewise, an item that appears relevant to experts may function poorly when administered to the intended population.

The broader process of evaluating questionnaire validity should therefore be distinguished from obtaining favorable expert ratings.

Comparing AI Feedback With Genuine Expert Content Review

The AI assessment and expert ratings point toward the same conceptual concern in this hypothetical example. However, they do not constitute the same kind of evidence.

The AI output served as preliminary feedback. The CVI summarized judgments actually obtained from reviewers, while the subsequent decision considered both the ratings and the theoretical definition.

Common Misconceptions About AI and Expert Content Validation

Use AI Before and Around Expert Review, Not as a Fictional Expert Panel

AI is most useful when it helps researchers prepare a stronger draft and manage the evidence obtained through a genuine review process.

Begin with a clear construct definition and an item-to-domain mapping. Use AI to flag possible weaknesses, then invite reviewers whose qualifications are relevant to the instrument's content and intended population.

Record reviewer qualifications, review instructions, ratings, comments, and substantive revisions. When AI contributes to the process, describe its role accurately.

The goal is to strengthen the evidence supporting questionnaire content, not to produce an impressive validation table without an accountable review process.

Before Reporting Expert Content-Validity Evidence

Frequently Asked Questions About AI and Expert Content Validation

AI Can Support Expert Review, but Cannot Manufacture Expert Evidence

How to Cite This Guide

This guide is intended to be read, shared, and used in research, teaching, and academic work. If you draw on its ideas, explanations, or other content, please acknowledge the source by citing the guide. Doing so gives appropriate credit and helps your readers locate the original resource.