03 · What You Need to Know
Building an AI-Assisted Questionnaire Development Process With Clear Evidence Boundaries
Generation and Validation Answer Different Questions
Questionnaire generation concerns developing the material respondents will encounter: instructions, items, response categories, and the organization of the instrument.
Validation concerns the evidence and reasoning supporting the interpretations and uses of the resulting responses or scores.
Questionnaire generation
What questions should be asked, how should they be worded, and how should respondents answer them?
Questionnaire validation
What evidence supports interpreting and using the responses or scores in the intended way?
These activities are connected, but they are not interchangeable. A well-designed item is a candidate for measurement. Its suitability must be evaluated in relation to the intended construct, population, and use.
The Standards for Educational and Psychological Testing emphasize that validity concerns the evidence supporting proposed score interpretations and uses. The quality of a questionnaire's wording is therefore not equivalent to the adequacy of its validity evidence.
Start With a Measurement Plan Before Using AI
The first methodological decision is whether the study requires a newly developed questionnaire, an existing instrument, or an adapted version of an established measure.
Where a suitable instrument already exists, its use may be preferable to developing a new scale without a clear justification.
If new development is warranted, specify the research objectives, intended respondents, variables, construct definitions, and proposed uses of the resulting data.
For latent constructs, the theoretical framework should guide item content. For factual or behavioral questions, the researcher should identify the information respondents can reasonably report.
AI can help create an initial questionnaire and translate objectives into candidate items, but the measurement plan should not be inferred solely from its output.
Use AI for Tasks It Can Support, Not Evidence It Cannot Supply
One practical way to maintain methodological clarity is to identify the contribution AI makes at each stage and distinguish that contribution from the evidence required to support the instrument.
| Activity |
Appropriate AI Assistance |
What AI Output Does Not Establish |
| Construct definition |
Organize supplied theoretical concepts and identify possible distinctions. |
That the proposed construct definition is theoretically justified. |
| Item generation |
Draft candidate questions from defined objectives and constructs. |
That the items adequately represent the intended domain. |
| Wording review |
Flag ambiguity, leading language, and multiple judgments. |
That actual respondents understand the items as intended. |
| Response design |
Suggest categories and identify apparent omissions. |
That respondents can use the categories accurately. |
| Content review |
Prepare review forms and organize genuine expert feedback. |
Independent expert agreement or content-validity evidence from fabricated ratings. |
| Pretesting |
Draft cognitive interview probes and test survey logic. |
How actual respondents interpret and answer questions. |
| Empirical evaluation |
Assist with statistical code and interpretation of authentic data. |
Validity or reliability without suitable observations and justified analyses. |
The table is not intended to prohibit AI use during later stages. Rather, it distinguishes analytical assistance from the provenance of the evidence being analyzed.
Follow a Defensible Development and Evaluation Sequence
Boateng and colleagues (2018) organize scale development into three broad phases: item development, scale development, and scale evaluation. Their framework includes domain identification, item generation, content assessment, pretesting, administration, item reduction, investigation of dimensionality, reliability evaluation, and validity assessment.
Artino and colleagues (2014) also describe a systematic process for developing questionnaires in educational research, emphasizing literature review, item development, expert review, cognitive interviewing, and pilot testing.
These frameworks illustrate why item generation is only one part of instrument development. The appropriate sequence may be iterative, and the procedures should be selected according to the questionnaire's purpose.
1. Define the measurement requirements
Establish the objectives, construct definitions, target population, and intended interpretation of responses or scores.
2. Generate and organize candidate items
Use AI where helpful to draft questions, suggest response formats, and prepare an item-to-construct mapping.
3. Conduct preliminary wording review
Examine ambiguity, leading questions, double-barreled items, missing response categories, and conceptual inconsistencies.
4. Obtain appropriate content evidence
Use qualified reviewers and, where relevant, intended respondents to evaluate the instrument's content and comprehensibility.
5. Pretest the questionnaire
Investigate how respondents understand and answer the questions and whether the administration procedures function appropriately.
6. Collect empirical measurement evidence
For scales requiring psychometric evaluation, examine relevant structural, reliability, and validity evidence using genuine response data.
7. Document the evidence and limitations
Explain the development decisions, AI contributions, evaluation procedures, findings, and remaining limitations.
The process should not be interpreted as a universal requirement that every questionnaire undergo factor analysis or the same statistical tests. Its purpose is to ensure that development activities are not mistaken for evidence they have not produced.
Why an AI Review Is Not the Same as Independent Validation
Suppose AI generates ten items measuring perceived usefulness and then evaluates whether those items are relevant to the construct.
The model may provide persuasive explanations, but its assessment is influenced by the definitions and assumptions used during generation. It may repeat conceptual errors embedded in the original prompt or overlook content that was never included.
Even when the AI system identifies genuine problems, the review does not establish how respondents interpret the items or whether their responses support the intended measurement model.
Independent expert review can contribute evidence about content relevance and coverage. Respondent pretesting can reveal comprehension and response-process problems. Empirical analyses can investigate measurement properties.
These sources of evidence address questions that cannot be resolved simply by asking the model whether its output is appropriate.
Do Not Confuse Content Review With Full Validation
Content review is an important part of instrument evaluation, particularly when developing a new scale. However, favorable content judgments do not automatically establish every aspect of validity.
For example, a panel of experts may agree that items are relevant to academic self-efficacy. Actual students may nevertheless interpret some items differently from the intended meaning.
Similarly, the item responses may not support the proposed dimensional structure when the scale is administered.
Researchers should distinguish expert content-validity review from the broader evidence needed to support score interpretation.
The necessary evidence should be determined by the proposed claims rather than by a desire to complete a fixed list of validation procedures.
Keep AI-Generated Data Separate From Genuine Evidence
AI can simulate respondents, generate hypothetical datasets, and produce numerical examples. These activities may be useful for testing questionnaire software or demonstrating analytical procedures.
However, synthetic responses should not be reported as data collected from the intended population.
For example, Cronbach's alpha calculated from AI-generated responses does not establish the internal consistency of scores obtained from actual participants. Likewise, factor loadings derived from synthetic data do not establish the empirical structure of the questionnaire in the target population.
Researchers can use AI-assisted software to analyze genuine data, but they must verify the procedures and clearly distinguish simulated demonstrations from empirical findings.
The same principle applies to expert ratings. A content-validity index calculated from invented reviewer scores does not represent independent expert agreement.
How Should You Prompt AI Without Asking It to Certify the Instrument?
A useful prompt identifies the specific development task and the evidence that remains unavailable.
Illustrative AI Prompt
Using AI With Explicit Methodological Boundaries
"I am developing a questionnaire measuring university instructors' perceived usefulness of generative AI. The construct is defined according to the theoretical framework supplied below. Review the candidate items for apparent construct alignment, unclear wording, overlapping content, and missing dimensions. Suggest revisions and explain their possible implications. Classify your observations as preliminary item-development feedback. Identify which questions require independent expert review, respondent pretesting, or empirical measurement evidence. Do not claim that the questionnaire is validated, simulate expert ratings as genuine data, or invent reliability and validity statistics."
This prompt does not prevent every possible AI error. It does, however, establish a useful distinction between preliminary observations and evidence that must be collected through other procedures.
How Should AI Assistance Be Documented?
Researchers should maintain a record of substantive AI contributions when they affect questionnaire development or analysis.
Relevant documentation may include the tool used, the purpose of the interaction, important prompts, generated candidate items, revisions adopted, and the researcher's reasons for accepting or rejecting suggestions.
The methodology section should distinguish these activities from the actual validation procedures.
For example, a researcher might report that generative AI assisted with preliminary wording review, followed by independent expert content assessment and cognitive interviews with members of the target population.
This is more informative than describing the questionnaire as "AI-validated," which does not identify the evidence supporting its intended use.
Researchers should also check applicable institutional, journal, and funder policies regarding AI use, confidentiality, and disclosure.
What Changes When an Existing Instrument Is Adapted?
Using an established questionnaire does not automatically eliminate the need to evaluate its suitability for a new population or context.
AI may help translate items, simplify terminology, or replace unfamiliar examples. However, these changes may affect the intended construct or how respondents interpret the questions.
Researchers should examine the original instrument's permissions, available measurement evidence, and relevance to the new population.
When substantial changes are made, additional content review, respondent pretesting, or empirical investigation may be warranted.
The central distinction remains unchanged: producing an adapted version is not the same as establishing evidence supporting its intended interpretation and use.
What Does Responsible AI-Assisted Questionnaire Development Ultimately Require?
Responsible practice does not require excluding AI from questionnaire development. It requires assigning AI an appropriate role and ensuring that claims about the instrument do not exceed the evidence obtained.
A researcher may use AI extensively during drafting while maintaining a rigorous evaluation process. Conversely, a questionnaire developed entirely without AI may still have serious validity problems if its constructs are poorly defined or its items are inadequately tested.
The methodological standard should therefore concern the quality of the instrument and the evidence supporting its use, not merely whether AI participated in its development.
