03 · What You Need to Know
What Makes a Research Question Suitable for Survey Data Collection?
Start with the Evidence You Need, Not the Questionnaire You Want to Distribute
A survey is a systematic approach to obtaining information from respondents through standardized questions or other structured requests for information. Depending on its design, it may collect quantitative responses, open-ended qualitative responses, or both.
The defining consideration is whether the information respondents provide can serve as appropriate evidence for your research question.
Suppose you want to investigate how university students perceive the usefulness of an artificial intelligence writing assistant. A survey may be appropriate because perceived usefulness concerns students' own judgments. They are relevant sources of information about those judgments.
Now suppose you want to establish whether the same assistant improves students' academic writing performance. Asking students whether their writing improved would measure perceived improvement, not necessarily actual improvement. You may need assessed writing samples, performance measures, or an appropriate comparative design.
The subject matter is similar, but the evidence required is different.
Surveys Are Particularly Useful for Describing Populations
Many survey studies seek to estimate the distribution of characteristics or responses within a defined population.
For example, researchers may investigate:
- What proportion of teachers report using generative AI for lesson preparation?
- How frequently do students report experiencing particular barriers to online learning?
- How do employees rate their perceived organizational support?
- What attitudes do researchers hold toward open research practices?
- How do reported experiences differ across demographic or professional groups?
These questions involve information that can be elicited systematically from respondents and compared across individuals.
However, estimating population characteristics requires more than asking standardized questions. The sampling frame, recruitment procedure, participation patterns, and treatment of missing data affect whether results can reasonably be generalized beyond the respondents.
A survey of volunteers may describe those volunteers accurately while providing a distorted picture of the broader population.
Surveys Can Examine Relationships, but Not Every Relationship Is Causal
Surveys are also useful when researchers want to examine associations among measured variables.
A study might investigate whether perceived instructor support is associated with student engagement, whether workload relates to reported burnout, or whether research experience is associated with attitudes toward open science.
Such relationships can be examined statistically when the variables are measured appropriately and the analysis matches the research question.
Nevertheless, an observed association may reflect confounding, reverse causation, measurement error, selection processes, or other explanations. A survey that measures exposure and outcome at the same time may have particular difficulty establishing which came first.
Surveys can form part of experimental or longitudinal investigations, so causal inference is not categorically impossible whenever survey instruments are used. The strength of a causal claim depends on the broader research design, not on the questionnaire itself.
Respondents Must Have Access to the Information You Are Asking For
Before selecting a survey, consider whether participants can reasonably know, remember, and communicate the requested information.
People may be reliable informants about some of their current preferences or subjective experiences. They may be less reliable when estimating infrequent past behaviors, recalling precise durations, identifying unconscious processes, or reporting actions affected by social expectations.
For example, asking teachers how many times they used a learning management system during the previous semester may produce approximate recollections. Platform activity records could provide more direct evidence of recorded system use, although those records would have their own limitations.
Similarly, asking students whether they collaborate during online activities may reveal their understanding of collaboration, but it may not accurately capture the frequency, quality, or structure of actual interactions.
The distinction is between what people report and what the researcher ultimately wants to establish.
The Construct Must Be Measurable Through Questions
Some research variables are relatively straightforward to ask about, such as educational attainment or whether a respondent has completed a particular training program. Others are latent constructs that cannot be observed directly, such as trust, motivation, perceived competence, or organizational commitment.
Measuring a latent construct requires a defensible operational definition and appropriate items or instruments. Researchers should consider content coverage, response processes, reliability, and validity evidence for the intended interpretation and population.
A convenient question is not automatically a valid measure.
For example, asking "Are you digitally competent?" may capture a respondent's broad self-assessment. It does not necessarily establish actual digital competence across the relevant knowledge and performance domains.
The decision about whether a construct requires multiple survey items should depend on its conceptual structure and the intended use of the resulting scores.
Surveys Work Best When Questions Can Be Understood Comparably
Standardization is one of the principal advantages of surveys. Respondents receive questions designed to elicit information that can be compared across participants.
Yet identical wording does not guarantee identical interpretation.
Terms such as "frequently," "effective," "regularly," and "high-quality" may mean different things to different respondents. Questions can also introduce ambiguity through unfamiliar terminology, double-barreled wording, vague reference periods, or response options that do not fit participants' circumstances.
Questionnaire development therefore requires more than grammatical correctness. Cognitive interviewing, expert assessment, pilot testing, and other forms of pretesting can help identify problems before full administration.
When respondents differ substantially in language proficiency, accessibility needs, or cultural interpretation, further adaptation and evaluation may be necessary.
Population Access and Sampling Are Part of Survey Suitability
A survey may be conceptually appropriate but practically unsuitable if the target population cannot be reached adequately.
Consider a study of informal workers who have limited internet access. An online questionnaire distributed through professional networking platforms may systematically exclude important parts of the population.
Before selecting a survey, define:
- The target population to which the research question refers.
- The population members who can realistically be identified or contacted.
- The sampling and recruitment approach.
- The likely coverage gaps and participation barriers.
- The implications of these limitations for inference.
The choice between a census and a sample survey affects whom you attempt to contact, but neither approach automatically eliminates nonresponse or measurement problems.
Likewise, selecting among online, paper, telephone, interviewer-administered, and mixed-mode surveys can influence who participates and how questions are answered.
Survey Timing Must Match the Phenomenon
If your objective is to describe a population at a particular point or period, a cross-sectional survey may be sufficient.
If you want to investigate change, development, or temporal ordering, repeated data collection may be necessary.
A single survey asking respondents to remember their attitudes before an intervention can sometimes provide retrospective information, but it is not equivalent to measuring those attitudes prospectively before and after the intervention.
The distinction between cross-sectional and longitudinal surveys therefore matters when change is central to the research question.
When Another Data-Collection Method May Be More Appropriate
Surveys are not necessarily the strongest source of evidence for every phenomenon involving people.
| Research objective |
Potentially appropriate approach |
Reason |
| Estimate the prevalence of reported attitudes or experiences |
Survey |
Standardized responses can support comparisons and population estimates when the sampling design permits. |
| Understand how participants interpret a complex experience |
Individual interviews |
Follow-up questions allow detailed exploration of meanings and circumstances. |
| Examine how views emerge through group discussion |
Focus groups |
Interaction among participants becomes part of the evidence. |
| Document what people actually do in a setting |
Direct observation |
Behavior can be examined without relying entirely on recollection or self-report. |
| Measure demonstrated knowledge or performance |
Assessment or performance task |
Demonstrated ability is observed rather than inferred solely from self-ratings. |
| Evaluate whether an intervention produces an effect |
Experimental or appropriately designed quasi-experimental study |
Comparison structures and temporal information can strengthen causal inference. |
| Analyze recorded events or transactions |
Existing administrative or digital records |
Records may capture relevant events more directly, subject to data-quality limitations. |
These approaches are not mutually exclusive. A study may use a survey to measure perceptions, observations to document behavior, and interviews to investigate why respondents interpret their experiences differently.
Multiple methods are useful when they contribute distinct evidence, not simply because a study appears more comprehensive with additional instruments.
Response Quality Can Undermine an Otherwise Appropriate Survey
Even when survey questions align with the research objective, responses may be affected by recall difficulties, social desirability, inattentive answering, question order, interviewer effects, or response-option design.
Nonresponse introduces another concern. People who participate may differ systematically from those who do not, particularly on variables relevant to the study.
A high response rate is desirable in many circumstances, but it is not conclusive evidence that nonresponse bias is absent. Conversely, a lower response rate does not establish the magnitude of bias without further assessment.
Survey suitability therefore includes whether the researcher can implement procedures that produce credible measurements and assess important sources of error.
Decisions about minimizing nonresponse before data collection should be made during planning rather than after participation proves disappointing.
Ethical and Practical Feasibility Matter
Survey participation may involve disclosure of sensitive experiences, personal information, employment circumstances, health information, or potentially identifiable combinations of responses.
Researchers should assess whether the information is necessary, whether participants can respond voluntarily, how privacy will be protected, and what institutional or legal requirements apply.
Power relationships require particular attention. Students surveyed by their instructors or employees surveyed by supervisors may perceive pressure to participate or provide favorable responses, even when the questionnaire states that participation is voluntary.
A feasible survey must therefore be methodologically appropriate and ethically defensible.
