01 · The Question
Can a Questionnaire Look Scientifically Sound While Measuring the Wrong Thing?
You ask generative AI to create a questionnaire measuring students' academic self-efficacy. Within seconds, it produces 20 statements, groups them into dimensions, assigns five-point response scales, and presents everything in a professional format.
Some statements concern students' confidence in completing academic tasks. Others ask whether they enjoy studying, work hard, or receive good grades. The questionnaire appears coherent because all the items relate to academic life.
Yet academic self-efficacy, motivation, enjoyment, and academic performance are not interchangeable. If these items are combined into one score, what exactly is being measured?
This is a particularly consequential risk of AI-assisted questionnaire development: a polished instrument can create an impression of measurement quality without adequately representing the intended construct.
03 · What You Need to Know
How an AI-Generated Questionnaire Can Measure the Wrong Construct
The Problem Begins When a Construct Is Treated as a Topic
A research topic describes an area of investigation. A construct is a defined attribute or concept that a researcher intends to measure.
For example, "students and academic success" is a broad research topic. Academic self-efficacy is a more specific construct concerning students' beliefs about their capabilities to perform academic tasks.
If a researcher asks AI to generate questions about academic success, the resulting questionnaire may include motivation, study habits, confidence, satisfaction, grades, and family support.
These concepts may all be relevant to academic success, but that does not make them indicators of one construct.
Boateng and colleagues (2018) emphasize defining the measurement domain before generating items. Their scale-development framework places domain identification and item generation before content evaluation and subsequent psychometric testing. Their methodological primer provides a detailed account of this sequence.
AI can help generate questions from research objectives, but researchers should not allow it to determine the construct's boundaries without theoretical justification.
What Is Construct Contamination?
Construct contamination occurs when a measure includes influences or content that do not belong to the attribute it is intended to represent.
Consider an academic self-efficacy scale containing the statement "I enjoy attending difficult classes."
Enjoyment may be associated with academic confidence, but it concerns a different psychological attribute. If the researcher combines enjoyment items with capability items, the resulting score may partly reflect attitudes toward studying rather than self-efficacy alone.
Another example is the statement "I always obtain high grades." This concerns reported achievement, not necessarily a student's belief about their capability to perform academic tasks.
The presence of related concepts does not automatically invalidate every item. The concern is whether the item content is justified by the intended construct definition and measurement model.
What Is Construct Underrepresentation?
Construct underrepresentation occurs when an instrument fails to cover important aspects of the intended measurement domain.
Suppose a researcher defines research self-efficacy broadly to include confidence in formulating research questions, designing studies, collecting data, analyzing evidence, and communicating findings.
AI generates ten items, but eight concern academic writing and two concern literature searching. The resulting questionnaire may contain relevant statements while inadequately representing the broader construct as defined.
This illustrates why relevance and comprehensiveness are different considerations. An item may belong to the construct without the complete item pool adequately covering its intended domain.
Construct contamination
The instrument includes content or influences outside the intended construct.
Construct underrepresentation
The instrument fails to capture important parts of the intended construct domain.
Both problems can exist in the same questionnaire. Adding more items will not necessarily correct either one unless the new content is theoretically appropriate.
How Can AI Produce Conceptually Misaligned Items?
Generative AI can produce fluent statements by drawing on associations among related concepts. When a prompt provides only a construct label, the system may include familiar topics associated with that label without maintaining the distinctions required for measurement.
| Intended Construct |
Potentially Misaligned Item |
What It May Measure Instead |
| Academic self-efficacy |
I enjoy completing difficult assignments. |
Academic enjoyment |
| Perceived usefulness of AI |
I use generative AI every day. |
Frequency of use |
| Research anxiety |
I have limited knowledge of statistical software. |
Self-reported knowledge |
| Behavioral intention |
I have used AI tools for several years. |
Previous behavior |
| Institutional support |
I am satisfied with my university. |
General institutional satisfaction |
These classifications are illustrative conceptual judgments. A particular item might have a defensible role in another instrument, but its inclusion must follow the intended definition rather than superficial similarity.
Why Does Professional Formatting Make the Problem Harder to Notice?
AI can produce questionnaires with consistent section headings, polished instructions, balanced-looking response scales, and explanations linking each item to a proposed dimension.
These features make the instrument easier to read and may encourage researchers to perceive it as methodologically complete.
However, formatting does not provide evidence that the content adequately represents the intended construct. Nor does the presence of a theoretical label above a group of statements establish that those statements measure that dimension.
A section titled "Perceived Usefulness" might contain items about ease of use, satisfaction, or behavioral intention. The label does not change what respondents are actually being asked.
The researcher should therefore evaluate each item's substantive meaning independently of its placement, presentation, or AI-generated explanation.
Can AI-Generated Items Drift Away From the Construct During Revision?
Yes. Conceptual misalignment can emerge during editing even when the original item was reasonably appropriate.
For example, an academic self-efficacy item might initially read "I can complete challenging academic assignments."
When asked to make the item more engaging, AI might suggest "I feel motivated to complete challenging academic assignments."
The revision changes perceived capability into motivation. Although the topic remains academic assignments, the psychological attribute has changed.
Similar problems may arise when AI is asked to improve questionnaire wording, translate items, or adapt an instrument for another population.
Researchers should compare original and revised items against the construct definition, not merely against each other.
Can High Reliability Hide Construct Problems?
Yes. Items may produce internally consistent responses without adequately representing the intended construct.
Suppose an AI-generated academic self-efficacy questionnaire contains many similar statements about enjoying academic work. Respondents may answer these statements consistently, producing a favorable internal consistency estimate.
That result would not establish that the scale measures academic self-efficacy. It might instead indicate consistency among enjoyment-related items.
Similarly, factor analysis may reveal a coherent response structure without proving that the factor corresponds to the construct named by the researcher.
The evaluation of questionnaire validity therefore requires conceptual justification alongside relevant empirical evidence.
How Can Researchers Detect Construct Misalignment Before Data Collection?
A useful approach is to develop a construct-to-item matrix. For each item, record the construct definition, intended dimension, observable indicator, and justification for including the statement.
Researchers can then examine whether each item represents the intended attribute, whether important dimensions are missing, and whether unrelated concepts have entered the item pool.
AI can help prepare this matrix and flag possible mismatches, but its assessment should be checked against the theoretical framework and relevant literature.
Illustrative AI Prompt
Screening a Questionnaire for Construct Misalignment
"I am developing a questionnaire measuring academic self-efficacy, defined as students' beliefs about their capability to perform specified academic tasks. Review the following items against this definition. For each item, identify the attribute it appears to measure, explain whether it represents academic self-efficacy or a related but distinct construct, and flag possible construct contamination or underrepresentation. Do not assume that an item is appropriate because it appears under a particular dimension heading. Identify uncertainties and suggest questions for qualified expert review. Do not claim that this screening establishes validity."
This prompt encourages AI to examine item meaning rather than accept the questionnaire's existing labels.
However, the model may still overlook theoretical distinctions or provide plausible but unsupported explanations. Its judgments should be treated as preliminary screening rather than final evidence.
What Evidence Is Needed After Conceptual Screening?
Conceptual alignment is an important starting point, but researchers may need additional evidence depending on the questionnaire's intended use.
Qualified reviewers can evaluate whether the item pool represents the intended construct and adequately covers its content domain. Cognitive interviewing can examine how respondents understand the items and arrive at their answers.
For multi-item scales, empirical analyses may investigate internal structure, reliability, and relationships with other variables. These findings should be interpreted in relation to the proposed construct and measurement model.
The Standards for Educational and Psychological Testing emphasize that validity concerns evidence supporting intended score interpretations and uses, rather than a property established through appearance or a single statistical result.
Researchers should not assume that a questionnaire has been validated merely because AI has approved its wording or because its items produce favorable statistics.
04 · A Practical Example
When an AI-Generated Academic Self-Efficacy Scale Measures Something Else
Hypothetical Example
A Professional-Looking Scale With Conceptual Problems
A researcher asks AI to generate an academic self-efficacy scale for university students. The model produces 15 items under the heading "Academic Self-Efficacy," all using a five-point agreement format.
Among the proposed items are "I can complete difficult assignments," "I enjoy studying challenging subjects," and "I usually obtain high grades."
Step 1: Examine the construct definition
The researcher identifies academic self-efficacy as perceived capability to perform academic tasks.
Step 2: Classify the generated items
The first item concerns capability, the second primarily concerns enjoyment, and the third concerns reported academic achievement.
Step 3: Identify construct contamination
The researcher recognizes that combining these items may produce a score influenced by enjoyment and achievement rather than self-efficacy alone.
Step 4: Examine construct coverage
The researcher reviews whether the remaining capability-focused items adequately represent the academic tasks included in the intended measurement domain.
Step 5: Revise and evaluate
Misaligned items are removed or reconsidered. The revised pool undergoes appropriate content review, respondent pretesting, and empirical evaluation.
The original questionnaire appeared complete because every statement concerned academic experiences. However, thematic similarity did not establish that the items represented one construct.
The researcher must establish what the scores are intended to mean before deciding whether the item pool is suitable.
06 · What This Means for You
Evaluate the Construct Before Evaluating the Questionnaire's Appearance
When reviewing an AI-generated questionnaire, begin with the construct definition and intended measurement domain. Do not rely on the model's headings, explanations, or apparent consistency as evidence that the items are appropriate.
Evaluate whether each item represents the intended attribute, whether related but distinct constructs have been introduced, and whether important content is missing.
A Simple Decision Framework
If an item represents a related but different construct
Remove it from the intended scale or justify its inclusion within a different measurement model.
If the item pool omits important content
Develop candidate items representing the missing aspects and evaluate their suitability.
If AI revises an established item
Compare the revision with the original construct definition and examine whether its meaning has changed.
If empirical statistics appear favorable
Interpret them alongside theoretical justification and other relevant validity evidence.
Keep a construct-to-item mapping and document the reasons for retaining, revising, or removing statements. Seek appropriate expert and respondent feedback before interpreting scores from a newly developed scale.
The central concern is whether the questionnaire measures what the researcher claims it measures, not whether the generated instrument looks professionally prepared.
