03 · What You Need to Know
How to Evaluate Outcome Assessment Procedures Before a Definitive Study
What Is Outcome Assessment Pilot Testing?
Outcome assessment pilot testing involves rehearsing the procedures used to obtain measurements of the outcomes specified in a research design.
The purpose is to determine whether the assessments can be administered, completed, recorded, and processed under conditions resembling those of the definitive study.
For example, a university research team may plan to measure student achievement before and after an instructional intervention. A pilot could examine whether the assessments fit the academic schedule, whether students understand the instructions, and whether the scores are captured correctly.
The focus is on the feasibility and integrity of the assessment process.
Outcome assessment piloting may also reveal questions requiring further psychometric evaluation, but it should not be treated as a substitute for that evaluation.
How Is Outcome Assessment Different From Outcome Measurement?
Outcome measurement concerns the construct, instrument, scoring, and interpretation of the outcome.
Outcome assessment procedures concern how the measurement is implemented within the study.
Outcome measurement
What is being measured, how scores are constructed, and what interpretations the resulting evidence can support.
Outcome assessment procedures
When, where, by whom, and through which operational processes the measurements are collected and recorded.
The distinction is useful because an instrument may have strong existing measurement evidence while its administration remains impractical in a particular setting.
Conversely, a questionnaire may be easy to administer but provide inadequate evidence for the intended construct interpretation.
When Is Outcome Assessment Piloting Particularly Important?
Piloting may be especially valuable when the study uses unfamiliar instruments, repeated assessments, complex scoring procedures, multiple assessors, or outcome measures requiring substantial participant effort.
It may also be important when measurements are collected remotely or across several institutions.
For example, an online educational trial may require students to complete assessments during scheduled learning activities. Technical interruptions, inconsistent instructions, or conflicting academic deadlines could affect completion.
A pilot can identify these operational risks before they affect the definitive study.
The need for testing should reflect the uncertainty and consequences of measurement failure rather than a universal requirement.
Which Assessment Procedures Should Be Tested?
| Procedure |
What to examine |
Potential problem |
| Participant instructions |
Whether participants understand how to complete the assessment |
Misinterpretation or inconsistent responses |
| Assessment timing |
Whether measurements can be obtained at intended time points |
Scheduling conflicts or missed assessments |
| Assessor training |
Whether assessors can administer and score procedures consistently |
Variation in administration or scoring |
| Assessment environment |
Whether conditions are sufficiently appropriate and comparable |
Distractions, technical problems, or uncontrolled assistance |
| Blinding |
Whether outcome assessors can remain unaware of assignments where required |
Knowledge of allocation may influence assessment. |
| Data completeness |
Whether required outcomes are recorded |
Missing or unusable measurements |
| Scoring procedures |
Whether scores are calculated and recorded correctly |
Incorrect coding or scoring |
| Participant burden |
Whether assessments are acceptable and practical |
Fatigue, withdrawal, or incomplete responses |
| Data transfer |
Whether assessment records reach the analysis dataset accurately |
Lost, duplicated, or incorrectly linked records |
The appropriate scope depends on the design. A single online questionnaire may require a narrower assessment pilot than a longitudinal trial involving several instruments and multiple assessors.
Should You Pilot the Primary Outcome Assessment First?
The primary outcome generally deserves particular attention because it is central to the definitive study's research question and statistical planning.
If the primary outcome cannot be collected reliably or with adequate completeness, the study may be unable to answer its principal question.
For example, an intervention trial may depend on a standardized achievement assessment administered at the end of the academic term.
A pilot can examine whether participants can complete the assessment within the intended window and whether the scoring procedure functions correctly.
However, secondary outcomes may also warrant intensive testing when they involve consequential risks or unfamiliar procedures.
Priority should follow the importance and uncertainty of each assessment.
How Should Assessment Timing Be Tested?
Timing can influence both feasibility and the meaning of outcome measurements.
For example, assessing academic engagement immediately after an intervention may produce different information from assessing engagement several weeks later.
A pilot can determine whether the proposed time points are practical and whether participants can complete assessments within the intended windows.
Researchers should also consider whether the timing is theoretically and empirically appropriate for the outcome.
However, a pilot cannot automatically establish the optimal time point for detecting an intervention effect, particularly when the sample is too small for reliable outcome comparisons.
Should You Test Repeated Outcome Assessments?
Yes, when the definitive study involves longitudinal or repeated measurements and their feasibility is uncertain.
Repeated assessments create additional demands involving scheduling, reminders, participant retention, record linkage, and consistent administration.
For example, students may complete a baseline engagement questionnaire but struggle to complete subsequent assessments during examination periods.
A pilot can identify whether the proposed schedule is manageable and whether missingness increases at particular time points.
Researchers should distinguish participants who withdraw from those who remain enrolled but miss specific assessments.
These circumstances have different implications for study planning and missing-data analysis.
How Should Data Completeness Be Evaluated?
Data completeness concerns the availability of required outcome measurements.
Researchers should define the relevant numerator, denominator, and assessment time point.
For example, if 36 of 40 enrolled participants provide usable primary outcome data, the observed completeness proportion is 90%.
However, this estimate does not guarantee equivalent completeness in the definitive study.
Researchers should examine why records are missing and whether the reasons suggest procedural problems, participant burden, or other conditions.
High completeness is useful feasibility evidence, but it does not independently establish measurement validity.
Should You Evaluate Missing-Data Patterns During the Pilot?
Yes. Missing data may arise from participant withdrawal, skipped items, technical failures, scheduling conflicts, or other causes.
For example, a questionnaire may have high overall completion but substantial missingness in items requiring detailed recall.
A pilot can help identify these patterns and inform revisions to administration procedures.
However, a small pilot may provide limited evidence about the missing-data mechanism that will operate in the definitive study.
Researchers should avoid assuming that missingness is random simply because no obvious pattern is observed.
The pilot can inform planning, but the definitive analysis still requires an appropriate strategy for handling missing data.
Should Assessor Training Be Piloted?
Yes, when assessments require judgment, standardized administration, or specialized skills.
For example, a study evaluating instructional performance may use trained observers to rate teaching practices.
A pilot can examine whether assessors understand the rubric, apply scoring rules consistently, and document relevant observations.
Training attendance alone does not establish assessment competence.
Researchers may need supervised practice, calibration exercises, or review of scoring discrepancies.
The appropriate procedures depend on the instrument and the intended interpretation of scores.
Can a Pilot Establish Inter-Rater Reliability?
A pilot may provide preliminary evidence about agreement or consistency among assessors, but the strength of that evidence depends on the number and range of observations, the scoring scale, and the reliability model.
For example, two observers may assign similar ratings to a small number of teaching demonstrations.
This may identify obvious scoring disagreements, but it does not necessarily provide a precise reliability estimate for the definitive study population.
Researchers should select appropriate statistics when quantitative agreement or reliability estimation is justified.
They should also distinguish agreement from reliability, as these are not interchangeable concepts.
Should Outcome Assessors Be Blinded?
Blinding may be important when knowledge of intervention assignment could influence outcome assessment.
For example, an observer rating classroom performance may unintentionally interpret behavior more favorably when aware that the class received a preferred intervention.
A pilot can examine whether assessors can remain unaware of allocation and whether intervention details are inadvertently disclosed.
However, blinding may be impractical for some educational interventions, particularly when the intervention is visible.
Researchers should consider alternative safeguards, such as standardized assessment procedures or independent scoring, when appropriate.
The inability to blind should be acknowledged rather than concealed.
How Should Blinding Feasibility Be Tested?
Researchers may examine whether allocation information is accessible to assessors, whether assessment materials reveal intervention status, and whether communication procedures protect blinding.
For example, an assessment file may contain a label identifying the intervention group even when the assessor is intended to be blinded.
A pilot can identify this problem before the definitive study.
However, asking assessors to guess allocation does not automatically provide a valid measure of blinding success, particularly when outcome-related information may influence guesses.
The evaluation should focus on the mechanisms by which allocation knowledge could arise and affect assessment.
Should Outcome Instruments Be Pretested Separately?
Sometimes. If the uncertainty concerns questionnaire wording, response categories, or comprehension, targeted instrument pretesting may be more appropriate than a comprehensive outcome assessment pilot.
For example, a newly developed student engagement scale may require cognitive interviewing and psychometric evaluation.
By contrast, an established instrument may require testing of administration timing, accessibility, or data capture in the proposed study.
These activities address different questions.
For questionnaire-specific considerations, see Should You Pilot Test Research Questionnaires Before the Main Study?.
Can a Pilot Establish Outcome Measure Validity?
Pilot testing can contribute evidence concerning administration and response processes, but it does not automatically establish validity.
Validity concerns the evidence and theory supporting the intended interpretation and use of scores.
For example, successful administration of a student engagement questionnaire does not establish that its scores adequately represent academic engagement in the intended population.
Additional evidence may be needed concerning content, internal structure, relationships with other variables, and other relevant considerations.
The Standards for Educational and Psychological Testing provide a framework for evaluating such evidence.
Researchers should specify what the pilot actually assessed rather than claim that the outcome measure was validated through successful administration.
Can a Pilot Determine Whether an Outcome Is Sensitive to Change?
A pilot may reveal obvious floor or ceiling effects, limited score variation, or practical difficulties in repeated measurement.
However, determining whether an outcome measure can detect meaningful change may require more extensive evidence than a small feasibility pilot can provide.
For example, a pilot may show that nearly all participants receive very high scores on an achievement assessment.
This may raise concerns about the instrument's ability to distinguish performance levels.
Nevertheless, a small sample may not adequately represent the distribution expected in the definitive study.
Researchers should interpret such findings as signals requiring investigation rather than definitive psychometric conclusions.
Should You Pilot Performance-Based Outcome Assessments?
Yes, when performance tasks involve unfamiliar administration, scoring, observation, or logistical requirements.
For example, a study may evaluate students' problem-solving skills through a practical task requiring several resources and trained assessors.
A pilot can examine whether instructions are clear, materials are available, tasks can be completed within the intended time, and scoring procedures are workable.
It may also reveal whether participants require accommodations or whether the assessment environment introduces avoidable variation.
However, a successful performance assessment pilot does not automatically establish the validity of the scoring rubric or the generalizability of performance scores.
How Should Digital Outcome Assessments Be Tested?
Digital assessments may depend on devices, internet connectivity, authentication, software configuration, and automated scoring.
Researchers should verify that the system records responses correctly and preserves the intended scoring and timing information.
For example, an online achievement assessment may function correctly on a desktop computer but display some items poorly on mobile devices.
A pilot can identify these technical barriers.
However, digital functionality does not establish that the assessment measures the intended construct appropriately.
Technical testing and measurement evaluation should be treated as related but distinct activities.
Can Assessment Burden Affect Research Findings?
Yes. Excessive assessment burden may contribute to incomplete responses, fatigue, withdrawal, or changes in participant behavior.
For example, students required to complete lengthy questionnaires immediately after demanding examinations may provide less considered responses or decline follow-up.
A pilot can assess completion time, perceived burden, and the practicality of assessment scheduling.
However, reducing burden should not compromise the information required to answer the research question.
Researchers should seek a proportionate balance between measurement requirements and participant demands.
Should Outcome Assessment Be Piloted Together With Intervention Delivery?
Sometimes. The interaction between intervention delivery and outcome assessment may create problems that isolated testing would miss.
For example, an intervention may require teachers to deliver feedback activities during the same period in which outcome assessments are scheduled.
A pilot can reveal whether the two processes compete for time or whether assessment procedures interfere with intervention implementation.
It may also identify risks of contamination or unblinding.
The relationship between these activities is discussed in Should You Pilot Test Intervention Delivery Before the Main Study?.
How Large Should an Outcome Assessment Pilot Be?
There is no universal sample size.
The appropriate scope depends on whether the objective is to identify administration problems, estimate completion rates, examine assessor consistency, or obtain preliminary information about score distributions.
A small pilot may reveal obvious technical or procedural difficulties but may not provide precise estimates of missingness or reliability.
Where outcomes are collected across clusters or sites, variation in assessment conditions may require attention to the number and diversity of settings.
The sample should be justified according to the feasibility questions rather than a fixed percentage of the definitive study.
What Progression Criteria Should Be Used?
Progression criteria may concern the proportion of usable primary outcome records, assessment completion within specified windows, assessor readiness, or the absence of critical procedural failures.
For example, a study may establish a minimum acceptable level of primary outcome completeness based on its analytic requirements.
However, thresholds should be justified by the study rather than adopted as universal standards.
Some problems, such as systematic scoring errors or compromised blinding, may be consequential even when overall completion is high.
Progression decisions should consider the nature of problems, uncertainty, and whether corrective modifications are feasible.
What If the Pilot Reveals Assessment Problems?
Researchers should identify the source and implications of each problem.
For example, missed assessments may reflect scheduling difficulties, while inconsistent scores may indicate inadequate assessor training or unclear rubric criteria.
The appropriate response depends on the problem.
Researchers may revise instructions, training, assessment windows, or technical procedures.
However, substantial changes to the outcome instrument or scoring system may affect measurement comparability and require further evaluation.
Consequential revisions should be documented and assessed against applicable ethics and protocol requirements.
How Should Outcome Assessment Pilot Findings Be Interpreted?
Findings should be interpreted as evidence about assessment feasibility under the observed conditions.
For example, a pilot may demonstrate that most participants completed the primary outcome assessment within the planned window.
This supports a conclusion about observed completion, not a guarantee of future data completeness.
Similarly, a pilot may identify scoring inconsistencies requiring additional assessor training.
It should not be treated as a definitive evaluation of intervention effectiveness merely because outcome scores were collected.
The distinction between feasibility evidence and unsupported conclusions is examined in How Do You Distinguish Useful Pilot Study Findings From Conclusions a Pilot Cannot Support?.
How Should Outcome Assessment Piloting Be Reported?
The report should describe the outcomes assessed, instruments used, administration procedures, timing, assessor training, and feasibility indicators.
Researchers should report relevant numerators, denominators, missingness, deviations, and corrective actions.
Where measurement properties were examined, the report should identify the specific evidence obtained and its limitations.
Researchers should distinguish successful administration from validity, reliability, and effectiveness conclusions.
When applicable, the CONSORT extension for randomized pilot and feasibility trials provides relevant reporting guidance.
Watch Out
Do not assume that a high assessment completion rate establishes measurement quality. Participants may complete every assessment while misunderstanding instructions, experiencing inconsistent conditions, or producing scores that do not adequately represent the intended construct.
