03 · What You Need to Know
How Questionnaire Length Influences Missingness and Data Quality
Why can longer questionnaires produce missing data?
Completing a questionnaire requires attention, comprehension, memory, and decisions about whether to disclose requested information. As these demands accumulate, some respondents may become fatigued, skip difficult items, provide less considered answers, or discontinue participation.
Survey methodology research has examined relationships between questionnaire length, response behavior, and completion. Findings suggest that burden can matter, although its effects vary across populations, survey modes, topics, and administration conditions.
Length is only one component of burden. Twenty technically demanding questions may require more effort than forty straightforward questions. A sensitive item can produce nonresponse even when it appears at the beginning of a brief questionnaire.
Researchers should therefore distinguish the number of questions from the actual cognitive, emotional, and practical demands of answering them.
What types of missingness might shortening reduce?
Questionnaire length can influence different forms of nonresponse, but these should not be treated as interchangeable.
| Problem |
What happens |
Potential contribution of shortening |
| Item nonresponse |
A participant skips one or more applicable questions. |
May reduce fatigue-related omissions but may not address sensitive or confusing questions. |
| Survey breakoff |
A participant begins but does not finish the questionnaire. |
May reduce accumulated burden that contributes to discontinuation. |
| Unit nonresponse |
An eligible participant does not participate in the questionnaire at all. |
A credible shorter time commitment may encourage some participation, but other barriers may dominate. |
| Low-quality responses |
Answers are recorded but may reflect inattention or insufficient effort. |
Reducing unnecessary burden may help, although response quality requires separate assessment. |
Shortening may reduce the amount of missing information without necessarily improving the quality of the remaining answers. Conversely, a participant may complete every item but provide unreliable responses.
Missingness and response quality should therefore be evaluated separately.
Is there an ideal number of questionnaire items?
No universally appropriate number of items applies across research settings.
A questionnaire containing 30 complex questions may require more time than one containing 60 simple questions. The appropriate length also depends on the participants, research purpose, administration mode, and whether established instruments must be administered in their validated form.
Researchers should estimate actual completion time through pilot testing rather than assuming that a particular item count is acceptable.
Completion-time estimates should reflect participants with different reading speeds, language proficiency, accessibility needs, and familiarity with the topic.
A stated completion time should be realistic. Underestimating it may damage trust and encourage respondents to abandon a questionnaire that takes substantially longer than expected.
Which questions should be considered for removal first?
Begin with questions that lack a clear scientific, analytical, ethical, or operational purpose.
Examples may include redundant background questions, repeated measurements of the same information without a methodological justification, or variables collected only because they might become interesting during analysis.
For each item, determine whether it supports the primary outcome, exposure measurement, essential confounding control, eligibility, measurement validation, or a justified secondary objective.
Some items may be retained because they contribute to a validated multi-item scale, even when individual items appear repetitive. Removing them can change the construct represented by the instrument.
The distinction between scientifically justified variable measurement and speculative data collection is useful when deciding what can be removed.
Shortening should follow a review of measurement purposes, not an arbitrary requirement to reduce the questionnaire by a fixed percentage.
Can removing items from a validated scale compromise validity?
Yes. Established scales are generally evaluated as particular combinations of items, response options, instructions, and scoring procedures.
Removing items may alter content coverage, reliability, factor structure, score interpretation, and comparability with previous studies.
For example, a twelve-item measure of academic motivation may represent several dimensions. Selecting only four items because they appear easiest could disproportionately represent one dimension while omitting others.
Researchers should first determine whether an officially developed or independently validated short form exists for the intended population and purpose.
If an instrument is modified, the revised version should not automatically be described as equivalent to the original. Appropriate evaluation may include expert content review, cognitive interviews, reliability analysis, and construct-validity evidence.
Permission or licensing conditions may also restrict modifications to particular instruments.
Watch Out
Do not shorten an established measurement instrument merely by deleting items with low response rates or weak statistical associations. Such decisions can change the construct being measured and may make previously established scoring and validity evidence inapplicable.
Why can question difficulty matter more than questionnaire length?
Respondents may skip questions because they cannot recall the requested information, do not understand the wording, or cannot identify a suitable response option.
A question asking students to estimate their average weekly internet expenditure over the previous year may be difficult even in a short survey.
Likewise, double-barreled questions, ambiguous reference periods, unfamiliar terminology, and complicated response matrices can increase the effort required to answer.
Instead of deleting a scientifically necessary question, researchers may improve its wording, specify the reference period, simplify the response format, or provide an appropriate explanation.
Cognitive interviewing can reveal how participants interpret questions and where misunderstandings occur. These findings may support improvements that reduce nonresponse without sacrificing content.
How does question sensitivity affect nonresponse?
Questions concerning finances, health, stigmatized experiences, or other private matters may generate nonresponse because participants are unwilling or unable to disclose the requested information.
Removing unrelated questions may reduce overall burden but will not necessarily resolve concerns about confidentiality or disclosure.
Researchers should explain why sensitive information is requested, apply appropriate privacy safeguards, and consider whether less intrusive measurements can adequately serve the scientific purpose.
For example, income categories may be more feasible than exact income amounts in some settings, although categorization sacrifices information and may not suit every analysis.
Response options such as "prefer not to answer" can respect participant autonomy and distinguish refusal from accidental omission. They do not eliminate missing information for the underlying variable.
Mandatory responses should be used cautiously and consistently with consent and ethical requirements.
Can question order affect missingness?
Yes. Questions appearing later in a questionnaire may be more vulnerable to breakoff-related missingness when participants become fatigued or run out of time.
However, moving every important question to the beginning is not always appropriate. Question order can affect interpretation through context, priming, and response effects.
For example, asking detailed questions about academic difficulties before a general academic confidence scale may influence how respondents interpret the confidence items.
Researchers should prioritize essential measurements while preserving a coherent questionnaire structure and any required instrument administration procedures.
When order effects are a concern, pilot testing or a randomized questionnaire-order experiment may provide useful evidence.
Can skip logic make questionnaires shorter without losing relevant information?
Conditional branching, commonly called skip logic, directs respondents only to questions applicable to their circumstances.
For example, participants who report never using a learning platform need not answer detailed questions about their experience using its specific features.
Appropriate skip logic can reduce unnecessary burden while retaining relevant information for eligible respondents.
However, incorrect branching can prevent participants from answering questions they should receive. Digital forms should therefore be tested for every major response pathway.
Researchers must also distinguish structurally inapplicable fields from genuinely missing answers. A skipped item that does not apply should not automatically be counted as item nonresponse.
Should large question matrices be shortened or redesigned?
Matrix questions present multiple statements using a common set of response options. They can make a questionnaire appear compact, but extensive matrices may require sustained attention and can be difficult to use on smaller screens.
Some respondents may select the same response repeatedly without carefully evaluating each statement, a pattern sometimes described as straightlining.
However, identical responses are not automatically evidence of poor-quality data. Participants may genuinely hold consistent views across related items.
Researchers should evaluate matrix usability, readability, and response patterns through pilot testing. Breaking a matrix into smaller groups may improve usability even when the total item count remains unchanged.
Changes to validated scales should still be evaluated for possible effects on measurement equivalence.
Can shorter questionnaires create new missing-data problems?
Yes. Removing variables may leave the study without information required for its primary analysis.
For example, shortening a baseline questionnaire by removing a measure of academic motivation could be consequential if motivation is a major confounder of voluntary tutoring participation and examination performance.
Similarly, eliminating questions that predict follow-up nonresponse may weaken the information available for missing-data analysis.
The resulting questionnaire might achieve a higher completion rate while producing a dataset less capable of supporting the causal question.
Researchers should therefore consider whether removing essential confounder measurements would undermine causal inference before finalizing reductions.
How should researchers test whether shortening actually helps?
Compare questionnaire versions using evidence rather than assuming that fewer items necessarily improve completion.
Where feasible, randomly assign pilot participants to a longer and shorter version. Record completion rates, item nonresponse, breakoff locations, completion time, and relevant indicators of response quality.
For established scales, evaluate whether the shortened version retains adequate measurement properties.
If participants are not randomly assigned to versions, differences in completion may reflect participant characteristics or administration conditions rather than questionnaire length.
Small pilots can reveal practical problems but may provide imprecise estimates of the effect of shortening.
The purpose of pretesting is to identify defensible improvements, not simply to demonstrate that the shortest version has the highest response rate.
What should be monitored during full data collection?
Researchers should monitor item-level nonresponse, survey breakoff, completion time, and patterns associated with participant characteristics or administration mode.
For online questionnaires, recorded progress information may help identify where participants discontinue. Such monitoring must be consistent with privacy notices and applicable data-protection requirements.
Unexpected missingness may reflect confusing questions, technical failures, inaccessible interfaces, or changes in participant circumstances.
Any revisions during collection should be documented because different questionnaire versions may produce measurements that are not directly comparable.
Questionnaire monitoring should be integrated with the assessment of variables most vulnerable to missing data and the broader missing-data prevention plan.
