03 · What You Need to Know
How Cost and Participant Burden Affect Measurement Decisions
What Makes a Research Measure Expensive or Burdensome?
The cost of an instrument extends beyond its purchase price. Researchers may need to pay for administration licenses, scoring services, equipment, software, training, translation, or qualified personnel.
Some instruments are inexpensive to acquire but costly to implement. A performance-based assessment may require individually supervised sessions, while a physiological measurement may involve laboratory facilities, equipment calibration, and specialist interpretation.
Participant burden refers to the demands placed on individuals through research participation. These may include time, cognitive effort, physical discomfort, emotional strain, travel, repeated assessments, and interference with ordinary responsibilities.
A measure can therefore be affordable to the researcher but burdensome to participants, or expensive to administer while requiring relatively little participant effort.
Researcher or institutional cost
Financial, staffing, equipment, training, administrative, and logistical resources needed to obtain and use a measure.
Participant burden
The time, effort, inconvenience, discomfort, or other demands experienced by people completing the measurement procedure.
Why Is the Most Psychometrically Impressive Measure Not Always the Best Practical Choice?
Measurement selection involves the intended interpretation of scores under actual study conditions. An instrument may demonstrate strong measurement properties in published evaluations but become unsuitable when the research team cannot reproduce its required administration procedures.
For example, a standardized assessment may require trained examiners and controlled testing conditions. Administering it without those conditions could undermine the relevance of its published reliability and validity evidence.
Similarly, a comprehensive questionnaire may cover the construct thoroughly, but its length may be unreasonable when participants must complete several other assessments during the same session.
The Standards for Educational and Psychological Testing emphasize that score interpretations and uses require appropriate supporting evidence. The practical implication is that feasibility should be evaluated alongside measurement quality, not treated as an unrelated administrative concern.
The decision should therefore concern the best-fitting measure for the intended research question under conditions that can actually be maintained.
How Can Participant Burden Affect Data Quality?
Excessive measurement demands may influence participation, completion, and response quality.
Long questionnaires can contribute to fatigue, reduced attention, skipped items, and response behaviors that do not reflect careful consideration. These effects are not inevitable, and their magnitude depends on questionnaire design, respondent characteristics, topic, motivation, and administration conditions.
Repeated assessments may create additional difficulties. Participants may withdraw, miss scheduled sessions, or provide incomplete responses when the study becomes difficult to accommodate.
Burden may also be distributed unequally. Participants with disabilities, caregiving responsibilities, limited internet access, or demanding work schedules may experience the same procedure differently.
Consequently, measurement burden can affect both the quality of the observed responses and the composition of the sample that remains available for analysis.
Should You Shorten a Comprehensive Instrument?
Not automatically. Removing items can reduce administration time, but it may also change what the instrument measures.
A questionnaire designed to assess several dimensions of a construct may lose important content if items are removed primarily to meet a preferred completion time. Shortening can also affect score reliability, structural validity, sensitivity, and comparability with previous studies.
Where available, an officially developed short form with relevant measurement evidence is generally preferable to an improvised subset of items.
However, a shorter instrument can be preferable when it adequately represents the intended construct and the additional content of the longer version offers little benefit for the specific research question.
The appropriate decision depends on the trade-off between measurement breadth and instrument length, rather than the assumption that more items necessarily produce better data.
Can You Reduce Costs Without Replacing the Instrument?
Before selecting another measure, identify which expenses are unavoidable and which arise from the proposed implementation.
Possible options include negotiating institutional access, applying for research funding, using an authorized academic license, sharing approved equipment, coordinating assessment sessions, or using a supported administration mode that requires fewer resources.
These options must remain consistent with the instrument's licensing conditions and administration requirements.
For example, an instrument that requires supervised administration should not automatically be moved online merely because remote delivery is cheaper. Evidence supporting equivalence between administration modes may be necessary.
Likewise, a copyrighted questionnaire should not be reproduced, modified, or distributed beyond its permissions simply because the research budget is limited.
How Should You Compare Feasible Alternatives?
Once unavoidable constraints are established, evaluate the remaining options systematically.
| Option |
Potential Advantage |
Important Limitation |
| Use the preferred instrument with additional resources |
Preserves the intended measurement strategy |
May require funding, equipment, permissions, or schedule changes |
| Use an established short form |
May reduce completion time and respondent burden |
May provide narrower construct coverage or different precision |
| Select another established instrument |
May provide an affordable and suitable alternative |
May assess different dimensions or reduce comparability |
| Use an alternative measurement method |
May better suit available resources or participant needs |
May require different validity evidence and interpretation |
| Reduce measurement frequency |
May lower repeated-assessment burden |
May weaken the ability to detect changes or characterize trajectories |
| Revise the research question or scope |
Aligns the study with what can be measured defensibly |
May reduce the breadth or ambition of the original investigation |
These options should not be evaluated solely through a numerical ranking. Some limitations may be unacceptable even when an alternative is inexpensive.
For example, a measure that does not assess the primary outcome cannot become appropriate merely because it is free.
When Can an Alternative Measurement Method Be More Practical?
Different methods may assess related aspects of a construct while imposing different costs and demands.
In educational research, learning engagement might be investigated through self-report questionnaires, classroom observations, learning-management-system records, or structured performance tasks.
These methods are not interchangeable. Self-report data may capture perceived engagement, whereas platform records may document observable interactions with a learning environment. Neither necessarily provides a complete account of the underlying construct.
Before replacing one method with another, determine whether the alternative still addresses the intended research question.
In particular, an apparently objective measure may be less appropriate than a carefully designed self-report measure when the construct concerns experiences, perceptions, or judgments that cannot be directly observed.
Should You Reduce the Number of Measurement Occasions?
Reducing assessment frequency can lower costs and burden, particularly in longitudinal or intervention studies. However, it may also change the research questions that can be answered.
Two measurement occasions may support an assessment of overall pretest-to-posttest change, but they generally provide less information about the trajectory of change than repeated observations across the intervention period.
Similarly, removing follow-up assessments may prevent researchers from investigating whether an observed effect persists.
Any reduction in measurement frequency should therefore be evaluated against the study's analytical objectives and expected timing of change.
What About Measuring Only a Subsample?
For expensive assessments, a planned subsampling strategy may sometimes preserve the opportunity to collect high-quality measurements while controlling costs.
For example, a researcher might administer an inexpensive questionnaire to the full sample and conduct a costly performance assessment among a probability-based subsample.
Such designs require careful planning. The subsample must be suitable for the intended analysis, and the resulting estimates may have greater uncertainty. Selection procedures and missing-data assumptions also matter.
Subsampling should not be introduced casually after recruitment simply because the assessment budget has been exhausted.
How Do You Know When a Compromise Has Gone Too Far?
A measurement compromise becomes problematic when it substantially weakens the link between the observed data and the intended construct or research inference.
Warning signs include replacing the primary outcome with a loosely related proxy, removing essential construct dimensions, using unqualified assessors, disregarding required administration procedures, or relying on an instrument whose score interpretation is unsupported in the target population.
When no suitable measure is available for the study population, additional instrument evaluation or development may be necessary.
A study with a narrower but defensible outcome can be more informative than one claiming to measure a complex construct using indicators that do not adequately represent it.
Watch Out
Do not solve a measurement feasibility problem by silently changing the construct. If a cheaper alternative measures confidence rather than competence, or frequency rather than quality, the resulting research question and conclusions must reflect that difference.
