03 · What You Need to Know
How Moving From Paper to Online Administration Can Affect Questionnaire Measurement
What Is a Questionnaire Administration Mode?
Administration mode refers to the method through which respondents receive and complete a questionnaire.
Common modes include paper self-administration, web-based self-administration, telephone interviewing, and interviewer-administered surveys.
Even within online administration, respondents may use desktop computers, tablets, or mobile phones.
These modes can differ in presentation, privacy, interviewer involvement, navigation, and the effort required to provide responses.
Consequently, administration mode is part of the measurement procedure rather than merely the method used to store responses.
What Is the Difference Between Electronic Conversion and Instrument Modification?
A faithful electronic conversion attempts to reproduce the original questionnaire's measurement task while changing the delivery medium.
A substantive modification changes features that may influence item meaning, response processes, or scoring.
Faithful electronic conversion
Preserves item content, instructions, response categories, reference periods, and scoring while adapting presentation to an electronic interface.
Substantive electronic modification
Changes questionnaire content or response behavior through altered wording, navigation, response requirements, scoring, or other consequential features.
The distinction is not always clear-cut. A change made for technical reasons may have measurement consequences.
For example, converting a printed matrix into separate mobile screens may preserve the words but change how respondents compare items and response categories.
Why Might Researchers Prefer Online Administration?
Online questionnaires may reduce manual data entry, facilitate geographically distributed recruitment, and support automated routing or scoring.
They may also allow participants to complete surveys at convenient times and support certain accessibility features.
However, these advantages depend on the platform, participants, and research setting.
Online administration may introduce difficulties involving internet access, device compatibility, technical literacy, privacy, and incomplete participation.
The choice should therefore reflect both research feasibility and measurement quality.
What Are Questionnaire Mode Effects?
Mode effects occur when the method of administration influences observed responses or other aspects of the measurement process.
For example, respondents may answer sensitive questions differently when completing a private online survey than when responding in the presence of another person.
Likewise, displaying all items together may encourage comparisons between questions that are less apparent when each item appears on a separate screen.
Mode effects can arise from presentation, perceived anonymity, interviewer presence, navigation, or other features.
However, not every difference between paper and online responses is necessarily caused by administration mode. Differences in participant characteristics and recruitment procedures can also contribute.
Which Changes Are Most Likely to Affect Measurement?
| Electronic Design Feature |
Potential Measurement Concern |
Evaluation Priority |
| Item wording |
Unintended changes in meaning |
Compare against the original instrument |
| Response labels |
Changed category interpretation |
Verify exact labels and ordering |
| Matrix presentation |
Different item comparison or response behavior |
Test across device sizes |
| One-item-per-page display |
Altered context and navigation |
Investigate response processes |
| Mandatory responses |
Forced answers to inapplicable or sensitive questions |
Review missing-response rules |
| Automated skip logic |
Incorrect routing or missing items |
Test every relevant response pathway |
| Mobile interface |
Hidden labels, scrolling, or accidental selections |
Conduct device-specific usability testing |
| Automated scoring |
Incorrect coding or reverse scoring |
Verify calculations against known examples |
| Privacy and anonymity |
Changed willingness to disclose information |
Evaluate administration conditions |
The importance of each feature depends on the instrument and the research question. A technical problem affecting a primary outcome deserves greater attention than a cosmetic difference with no plausible measurement consequence.
Does Reproducing the Exact Questions Guarantee Equivalence?
No. Identical wording helps preserve content but does not guarantee identical response processes.
For example, a paper questionnaire may display instructions and several items together. In an online version, instructions may disappear when respondents move to another page.
Participants might then answer later questions without remembering the intended reference period.
Similarly, response labels that remain visible on paper may be obscured by scrolling on a mobile device.
Researchers should therefore examine the complete respondent experience rather than compare wording alone.
Can Online Administration Change Social Desirability Bias?
Potentially. Respondents' willingness to disclose sensitive information may depend on perceived privacy, anonymity, and the presence of other people.
For example, students may respond differently to questions about academic misconduct when completing a survey privately than when answering in a supervised classroom.
However, online administration does not automatically guarantee greater honesty.
Participants may remain concerned about identification, institutional monitoring, or the handling of their data.
The researcher should distinguish actual confidentiality protections from participants' perceptions of privacy.
Does Online Administration Automatically Improve Data Quality?
No. Electronic collection can reduce transcription errors and support programmed checks, but it may introduce other sources of error.
For example, poorly configured forms may record incorrect values, permit duplicate submissions, or route respondents past required questions.
Automated validation can also create problems when legitimate answers are rejected or participants are forced to choose inaccurate responses.
Researchers should evaluate the complete data collection process, including recruitment, participation, item display, response recording, and scoring.
The absence of manual data entry does not establish the absence of measurement error.
Should Every Online Questionnaire Require Responses to All Items?
Not necessarily. Mandatory-response settings may reduce item-level missingness, but they can also force respondents to answer questions they cannot meaningfully complete.
For example, a participant who has never used a particular learning technology may be unable to rate the quality of feedback received through it.
If the survey provides no appropriate non-applicable response, the participant may select an arbitrary category or abandon the questionnaire.
Researchers should preserve the original instrument's response rules where appropriate and consider the ethical and measurement implications of forced responses.
Missing data should not automatically be treated as a technical defect that must be prevented at any cost.
Can Mobile Administration Affect Questionnaire Responses?
Yes. Mobile devices introduce distinct interface constraints, including smaller screens, touch-based selection, and additional scrolling.
For example, a seven-category matrix may fit comfortably on a desktop monitor but require horizontal scrolling on a phone.
Respondents may not see the full category labels while selecting an answer.
Such problems can affect response accuracy and completion.
Researchers should test the electronic questionnaire on the devices participants are likely to use rather than assuming that desktop functionality establishes mobile usability.
Should Matrix Questions Be Converted Into Individual Items?
They can be, particularly when the original matrix is difficult to use on small screens.
However, the change may alter item context, response patterns, and the way respondents compare statements.
A matrix can make common response options visible across multiple items, while individual screens may encourage respondents to consider each item separately.
Neither format is universally superior.
The decision should consider usability, response-process evidence, and the instrument's original presentation.
When a substantial layout change is necessary, researchers should evaluate whether the modified format continues to support the intended measurement.
Can Automated Skip Logic Change the Instrument?
Yes, particularly when the routing differs from the original questionnaire.
Skip logic determines which questions respondents encounter based on previous answers.
For example, an online survey may automatically skip follow-up questions when a respondent indicates that an activity does not apply.
This may improve efficiency when it faithfully implements the intended questionnaire instructions.
However, incorrect routing can systematically omit relevant items or produce inconsistent data.
Researchers should test each relevant pathway and verify that the electronic logic reproduces the intended administration procedure.
What Happens When a Questionnaire Uses Reverse-Scored Items?
Electronic platforms may automatically calculate reverse-scored values, but the calculations must be verified.
For a five-category item coded from 1 to 5, reverse scoring ordinarily maps 1 to 5, 2 to 4, and 3 to 3.
However, an instrument using codes from 0 to 4 requires a different transformation.
Incorrect coding can systematically distort total and subscale scores even when participants answer every question correctly.
Researchers should test scoring algorithms using known response patterns before deploying the survey.
The principles for reverse-worded and reverse-scored questionnaire items remain applicable regardless of administration mode.
Does Online Conversion Require a New Validation Study?
Not universally. The necessary evidence depends on how faithfully the instrument is reproduced and how consequential the intended use is.
The ISPOR ePRO Task Force report by Coons and colleagues distinguishes levels of modification when migrating patient-reported outcome measures from paper to electronic administration.
Its recommendations support proportionate evaluation based on the magnitude of the change, rather than assuming that every electronic conversion requires identical testing.
However, the framework was developed for patient-reported outcome measures and should not be applied mechanically to every educational or social research instrument.
A faithful electronic conversion may require focused usability and equivalence evaluation, while substantial changes to wording, response options, or scoring may require broader measurement evidence.
What Does Research Say About Paper and Electronic Questionnaire Equivalence?
Research has examined whether electronic and paper versions of established measures produce comparable scores.
Gwaltney, Shields, and Shiffman's meta-analytic review of electronic and paper patient-reported outcome measures found evidence supporting comparability across the studies examined.
However, such findings should not be interpreted as proof that every online questionnaire is equivalent to its paper version.
Comparability depends on the instrument, conversion procedures, respondent population, device, and administration conditions.
The relevant conclusion is that electronic conversion can preserve measurement properties under suitable conditions, not that mode effects are impossible.
How Can Researchers Evaluate Mode Equivalence?
Evaluation should address the aspects of equivalence relevant to the intended use.
Researchers may compare respondent comprehension, item distributions, reliability, dimensional structure, and score agreement across paper and online versions.
When feasible, randomized assignment to administration modes can help distinguish mode-related differences from differences in participant characteristics.
Within-person comparisons may also be informative but require attention to order, memory, and carryover effects.
Statistical similarity alone is not sufficient. For example, similar mean scores may conceal differences in how particular items function.
Likewise, a high correlation between versions does not establish that their scores are numerically interchangeable.
Should the Same Participants Complete Both Paper and Online Versions?
They may, depending on the evaluation design.
A within-person comparison can examine score agreement while controlling for stable individual differences.
However, completing one version may influence responses to the second because participants remember their previous answers or become familiar with the questions.
Changes in the underlying construct between administrations may also affect the comparison.
Researchers should consider counterbalancing, suitable intervals, and the stability of the measured attribute.
There is no universally superior comparison design. The appropriate approach depends on the instrument and the specific equivalence question.
Can Paper and Online Responses Be Combined in One Dataset?
Potentially, when sufficient evidence supports the intended comparability.
However, combining responses from different modes without examining mode-related differences may introduce heterogeneity that is difficult to interpret.
For example, if students completing paper questionnaires do so in supervised classrooms while online respondents participate independently, differences may reflect both administration mode and setting.
Researchers should document the mode used by each participant and consider whether relevant differences require evaluation or analytical adjustment.
Where the research depends on comparable latent scores, appropriate measurement invariance or other equivalence analyses may be informative.
Adjustment alone cannot guarantee equivalence when important response-process differences remain unresolved.
Does Changing Administration Mode Affect Sampling?
It can. Online recruitment may reach different participants from paper-based recruitment.
For example, students with reliable internet access may be more likely to complete a web questionnaire, while those with limited connectivity may be underrepresented.
These differences concern coverage and selection, which are distinct from measurement mode effects.
Even if the questionnaire functions identically for participants who complete it, the resulting sample may differ from the population the researcher intended to represent.
Researchers should therefore evaluate both the quality of the electronic measurement and the representativeness of the recruitment process.
What Accessibility Issues Should Be Considered?
Electronic questionnaires should be usable by participants with relevant disabilities and access needs.
Potential concerns include keyboard navigation, screen-reader compatibility, readable labels, adequate contrast, and the accessibility of response controls.
For example, a custom visual rating slider may be difficult to use with assistive technology even when the underlying question is conceptually appropriate.
Researchers should select accessible controls and test the questionnaire under realistic conditions.
Accessibility modifications should preserve the intended response task as far as possible, with additional evaluation when necessary.
Improved accessibility is not merely a technical convenience. It can affect who is able to participate and how accurately they can respond.
What Privacy and Data Protection Issues Arise?
Online questionnaires may collect or process information through external platforms, servers, and connected services.
Researchers should examine how responses are transmitted, stored, accessed, retained, and potentially linked to identifying information.
For example, a survey advertised as anonymous should not be described that way without considering whether the platform records identifiable information such as account credentials or other metadata.
Applicable institutional ethics requirements, data protection obligations, and platform agreements should be considered.
In the Philippines, the Data Privacy Act of 2012, Republic Act No. 10173, may be relevant when personal information is processed, alongside applicable regulations and institutional policies.
Compliance requirements depend on the research activities and should not be inferred solely from the fact that a questionnaire is administered online.
How Should Online Questionnaires Be Pretested?
Pretesting should examine both the instrument's measurement content and the electronic system used to administer it.
Researchers should verify that all questions, instructions, response categories, and scoring rules match the intended version.
Test the survey on representative devices and browsers, including mobile interfaces when relevant.
Check navigation, skip logic, missing-response behavior, progress indicators, submission confirmation, and data export.
Appropriate cognitive interviewing can reveal whether interface changes affect how respondents interpret and answer questions.
A technical test by the research team is useful but should not automatically replace evaluation with intended respondents.
What Reporting Standards Are Relevant to Online Surveys?
The Checklist for Reporting Results of Internet E-Surveys, commonly known as CHERRIES, provides guidance on reporting aspects of internet-based surveys.
Its topics include survey development, recruitment, administration, response rates, and measures intended to address particular data-quality concerns.
However, CHERRIES is a reporting framework rather than a psychometric validation procedure.
Following a web-survey reporting checklist does not establish that an electronic version of a paper questionnaire is measurement-equivalent.
Researchers should use reporting guidance alongside appropriate instrument-specific evaluation.
How Should a Paper-to-Online Conversion Be Documented?
Describe the original instrument and the electronic version administered.
Document the platform, item presentation, response options, navigation, required-response settings, scoring, and relevant device conditions.
Explain any changes from the paper version and identify the evidence supporting their acceptability.
If paper and online scores are combined or compared, report the basis for assuming or evaluating comparability.
Preserve version information and relevant testing records as part of documenting research instrument modifications.
When Should Researchers Avoid Online Conversion?
Online conversion may be inappropriate when the electronic format cannot adequately reproduce essential features of the measurement task.
For example, a paper assessment requiring detailed spatial annotation or specialized materials may not transfer meaningfully to a basic survey interface.
It may also be unsuitable when participants lack reliable access to compatible devices or when the proposed platform cannot meet necessary privacy and accessibility requirements.
Researchers should consider whether an alternative digital implementation, mixed-mode design, or retention of paper administration would better preserve measurement quality.
The decision should reflect the research purpose and participants' circumstances rather than an assumption that digital administration is inherently preferable.
Watch Out
Do not assume that copying the wording of a paper questionnaire into an online form guarantees equivalence. Changes in navigation, response requirements, device presentation, and administration conditions may affect how participants answer, even when the text remains identical.
