03 · What You Need to Know
Where Generative AI Fits in Questionnaire Development
Start With What You Need to Measure, Not With an AI Prompt
A questionnaire should be developed from the research problem, objectives, relevant literature, and the information required to answer the research questions. If the questionnaire measures a psychological, educational, or behavioral construct, its theoretical definition and dimensions also need to be established.
Consider a researcher investigating teachers' acceptance of generative AI in classroom instruction. Asking an AI system to create 20 questions about teachers and AI might produce items about confidence, perceived benefits, institutional support, frequency of use, and ethical concerns.
All these topics may be interesting, but they are not necessarily dimensions of the same construct. Unless the researcher specifies the conceptual framework, the system may combine theoretically distinct concepts into one questionnaire without explaining their relationships.
A stronger starting point is to identify the constructs and operational definitions before requesting items. For example, a study based on the Technology Acceptance Model might distinguish perceived usefulness from perceived ease of use. The researcher should define what each means in the particular study rather than allowing the model to decide which dimensions belong in the instrument.
AI can help translate these definitions into candidate questions, but the conceptual decisions should remain grounded in theory and the study's objectives.
What Can AI Actually Do During Questionnaire Development?
Generative AI can support several activities that would otherwise involve repeated drafting and editing. Its usefulness is generally greatest when the researcher supplies a clear framework and evaluates the suggestions rather than accepting the output wholesale.
| Development Activity |
Potential AI Contribution |
What the Researcher Must Verify |
| Planning the questionnaire |
Suggest sections and organize topics. |
Whether every section is relevant to the research objectives. |
| Generating items |
Draft candidate questions from supplied constructs or objectives. |
Whether each question represents the intended concept. |
| Refining wording |
Suggest clearer, shorter, or more neutral alternatives. |
Whether revisions preserve the intended meaning. |
| Designing responses |
Propose agreement scales, frequency categories, or other response formats. |
Whether the response options fit the question and population. |
| Reviewing the draft |
Flag possible ambiguity, repetition, or leading language. |
Whether the identified problems are genuine and whether others were missed. |
| Preparing for pretesting |
Suggest interview probes and review questions. |
Whether actual respondents understand the items as intended. |
These activities concern drafting and preliminary review. They should not be mistaken for empirical evidence that a questionnaire performs adequately.
How Should You Prompt AI to Generate Questionnaire Items?
A useful prompt supplies the information that a questionnaire developer would ordinarily need: the research objective, target respondents, construct definitions, relevant dimensions, response format, and any contextual restrictions.
For instance, rather than requesting a questionnaire about educational technology, you could provide the following instructions:
Illustrative AI Prompt
Generating Candidate Questionnaire Items
"I am developing a questionnaire for university instructors who have experience using generative AI in teaching. My study examines perceived usefulness, defined as the extent to which instructors believe generative AI improves their teaching performance. Suggest six candidate statements measuring this construct. Use simple language, focus each statement on one idea, avoid assuming that AI is beneficial, and propose a consistent five-point agreement scale. Explain how each statement relates to the supplied definition. Do not claim that the items are validated."
This prompt narrows the task to a defined construct and makes the intended population explicit. It also asks the system to justify the proposed items, which gives the researcher something concrete to evaluate.
The resulting statements would still require review. Researchers can examine the relationship between their research objectives and proposed survey questions before deciding which items deserve further development.
AI Can Produce Questions Without Understanding the Measurement Problem
Generative AI produces responses based on learned patterns and the information supplied in the interaction. It does not independently establish that a questionnaire adequately represents a construct, that respondents will interpret its items consistently, or that the resulting scores will support the intended interpretation.
For example, an AI-generated questionnaire about digital literacy might emphasize familiarity with applications and frequency of technology use. Yet digital literacy may also involve evaluating information, exercising judgment, and understanding digital risks, depending on the adopted framework.
A questionnaire that overlooks these dimensions could appear coherent while providing incomplete construct coverage. This is especially consequential when the researcher intends to calculate a total score or compare groups.
Questionnaire generation
Producing candidate questions, instructions, response options, and an organized draft instrument.
Questionnaire validation
Collecting and evaluating evidence that supports the intended interpretation and use of questionnaire responses or scores.
A system may perform the first activity convincingly without establishing the second. The distinction is fundamental to responsible AI-assisted instrument development.
Different Questionnaires Require Different Types of Evidence
Not every questionnaire is a psychometric scale. Some questionnaires collect factual information, such as teaching experience, device ownership, or the number of professional development activities attended. Others assess attitudes, perceptions, motivations, or latent constructs that cannot be observed directly.
Both types require careful design, but the evidence needed to support their use differs.
A descriptive questionnaire should be evaluated for clarity, relevance, response coverage, recall demands, and the likelihood that respondents can provide accurate answers. A multi-item scale may additionally require evidence concerning content validity, internal structure, reliability, and relationships with other variables, depending on its intended use.
For measurement instruments, the COSMIN framework emphasizes content relevance, comprehensiveness, and comprehensibility. Although COSMIN was developed primarily for health-related outcome measures, these considerations are informative for questionnaire development in other fields.
Consequently, researchers should avoid treating every questionnaire as if it requires identical statistical procedures. They should also avoid treating a questionnaire as valid merely because its items appear sensible.
Can AI Help Identify Poorly Written Questions?
Yes. AI can provide a useful preliminary review of item wording. It may flag questions that suggest a preferred answer, combine multiple ideas, contain vague terms, or use unnecessarily complicated language.
For example, the question "How satisfied are you with the excellent AI tools provided by your university?" assumes that the tools are excellent. A more neutral formulation would ask respondents how satisfied or dissatisfied they are with the AI tools provided.
Likewise, "AI helps me prepare lessons and assess students effectively" combines two activities that a respondent may evaluate differently. The researcher could separate lesson preparation from student assessment.
AI-assisted reviews can be useful for identifying leading questions and double-barreled items. However, the absence of an AI warning does not establish that the wording is acceptable. A question can be grammatically clear while remaining conceptually inappropriate.
Why Human Review and Pretesting Remain Necessary
Questionnaire development is an iterative process. Drafting is followed by review, testing, revision, and, when appropriate, psychometric evaluation.
The American Association for Public Opinion Research recommends testing questionnaires with people similar to the intended respondents. Cognitive interviewing can reveal how participants understand a question, retrieve relevant information, and decide on an answer. Pilot testing can also identify problems with questionnaire administration and data collection procedures.
These procedures provide information that cannot be established simply by asking an AI system whether the questionnaire is understandable.
For instruments measuring latent constructs, expert review and appropriate empirical analyses may be needed to examine the adequacy of the items and their measurement properties. AI can help organize reviewer feedback or suggest revisions, but it cannot substitute for evidence obtained through suitable evaluation procedures.
In particular, evaluating questionnaire validity involves more than asking a model to rate items or calculate an apparent content-validity score from its own judgments.
What About Confidentiality, Copyright, and Existing Instruments?
Researchers should also consider what information they share with an AI service. Unpublished research materials, confidential participant information, restricted institutional documents, and sensitive study details should not be entered into an external system without checking the applicable privacy requirements, institutional approvals, and service terms.
Existing questionnaires require particular care. A published instrument may be copyrighted, licensed, or subject to conditions governing modification and redistribution. Asking AI to reproduce or rewrite an established scale does not remove those obligations.
When a suitable instrument already exists, evaluating whether it can be used or appropriately adapted may be methodologically preferable to generating a new one. Researchers should verify the original source, permissions, and available validity evidence rather than assuming that AI-generated alternatives are equivalent.
04 · A Practical Example
Using AI to Draft a Questionnaire About Teachers' AI Acceptance
Hypothetical Example
Developing Candidate Items for Perceived Usefulness
A researcher wants to investigate university instructors' acceptance of generative AI for teaching. The study draws on the Technology Acceptance Model and includes perceived usefulness as one of its constructs.
The researcher defines perceived usefulness in this context as the belief that using generative AI can improve teaching-related performance. Rather than requesting an entire questionnaire without guidance, the researcher asks AI to propose candidate items specifically for this construct.
Step 1: Define the construct
The researcher establishes the construct definition using relevant theoretical literature and identifies the intended respondents as university instructors.
Step 2: Request candidate items
AI proposes statements such as "Using generative AI helps me prepare teaching materials more efficiently" and "Using generative AI improves my effectiveness in completing teaching-related tasks."
Step 3: Evaluate conceptual alignment
The researcher examines whether each statement represents perceived usefulness. An AI suggestion such as "I enjoy experimenting with generative AI" is set aside because enjoyment is not the same as perceived usefulness.
Step 4: Review wording and response options
The researcher considers whether respondents can interpret the statements consistently and whether an agreement scale is suitable for the intended measurement approach.
Step 5: Obtain evidence and revise
Candidate items undergo appropriate expert review, respondent pretesting, and further empirical evaluation where required. Items may be revised or removed based on the findings.
The initial AI output provides a starting pool of items, not a finished measurement instrument. Even an item that appears closely aligned with the theoretical definition may function poorly when actual respondents interpret or answer it.
The researcher should also document how the items were generated, which suggestions were retained, and what revisions were made. This creates a defensible development record rather than relying on an undocumented AI conversation.
06 · What This Means for You
How to Use AI Without Compromising Questionnaire Quality
The most defensible approach is to treat generative AI as a drafting and critical-review assistant. Its suggestions should enter the same methodological evaluation process as questions written by a researcher.
Begin by determining whether you need to develop a new questionnaire at all. Existing instruments may already measure your intended constructs and have relevant evidence supporting their use. If no suitable instrument exists, establish your conceptual framework and development plan before generating items.
A Simple Decision Framework
If an established questionnaire already measures your intended construct
Evaluate its suitability, available validity evidence, and permissions before creating an alternative.
If you are developing a new questionnaire
Define the constructs and information requirements first, then use AI to suggest candidate items for review.
If AI identifies potential wording problems
Examine the suggestions critically and test important revisions with appropriate respondents.
If the questionnaire will produce scores representing latent constructs
Plan suitable content review and empirical evaluation before interpreting those scores as measures of the intended constructs.
Keep a record of the AI tool used, the purpose of its assistance, important prompts, and substantive revisions when these details are relevant to methodological transparency. Check applicable institutional, journal, and funder policies regarding AI use and disclosure.
Most importantly, do not let the speed of question generation determine the pace of instrument evaluation. A questionnaire can be drafted in minutes, but evidence supporting its intended use requires a separate process.
