Which Research Procedures Should You Prioritize When Planning a Pilot Study?

Which Parts of Your Research Should You Test First?

You have developed a research design and decided that some preliminary testing would be useful. Your proposed study involves recruitment, informed consent, questionnaires, intervention delivery, outcome assessment, data management, and statistical analysis. Ideally, you would like to test everything before beginning the main investigation.

But time, funding, participant access, and research personnel are limited. Testing every procedure with equal intensity may be impractical, and some procedures are already well established.

Should you prioritize recruitment because the study cannot proceed without participants? Should you focus on outcome measurement because unusable data would undermine the findings? Or should you rehearse the entire research process to identify problems that individual pretests might miss?

The appropriate decision depends on the consequences of procedural failure, the uncertainty surrounding each procedure, and what evidence is already available.

Prioritize Procedures That Present the Greatest Threat to the Definitive Study

How to Identify and Prioritize Research Procedures for Pilot Testing

Why Should Pilot Procedures Be Prioritized?

A pilot study is intended to provide information that helps researchers decide whether and how a proposed definitive study can proceed.

It is not necessarily a comprehensive rehearsal of every research activity. Some pilots reproduce much of the intended study, while others focus on selected components that present particular uncertainties.

Prioritization helps direct limited resources toward procedures that could meaningfully affect recruitment, participant protection, implementation, data quality, or the validity of the eventual research conclusions.

For example, testing the formatting of an already established questionnaire may contribute little if the main unresolved problem is whether participants can complete repeated outcome assessments.

Conversely, concentrating exclusively on recruitment would be inadequate if the intervention cannot be delivered consistently.

The appropriate priority is therefore determined by the study's actual vulnerabilities rather than by a standard checklist applied without consideration of context.

What Makes a Research Procedure High Priority?

A procedure deserves particular attention when its failure would have serious consequences and there is insufficient evidence that it will work under the proposed conditions.

Three considerations are especially useful: the consequences of failure, uncertainty about feasibility, and the extent to which preliminary testing can inform a decision.

For example, an unfamiliar recruitment procedure may warrant piloting when the definitive study depends on enrolling a substantial number of participants within a limited period.

A complex outcome assessment may warrant attention when incomplete or inaccurate measurement would undermine the primary research question.

However, a serious ethical risk should not be assigned low priority simply because its occurrence appears unlikely. Participant safety, valid consent, and applicable research governance requirements remain essential regardless of an operational ranking.

How Can You Assess Procedural Risk?

A structured risk assessment can help researchers compare procedures without assuming that every uncertainty has equal importance.

Researchers may use qualitative judgments or a locally developed prioritization matrix. However, numerical risk scores should not be treated as validated measures unless an appropriate basis exists.

The purpose is to make decisions explicit and defensible, not to create an illusion of precision.

Should Recruitment Be a Priority?

Recruitment should receive high priority when uncertainty about identifying, contacting, screening, or enrolling participants could prevent the definitive study from achieving its required sample.

For example, a proposed educational intervention may depend on recruiting teachers from several institutions. Even when institutional administrators express support, individual teachers may decline participation because of workload or scheduling constraints.

A pilot can assess eligibility yield, response patterns, enrollment rates, and the resources required for recruitment.

However, recruitment need not dominate every pilot. A study using an established participant registry and previously demonstrated procedures may have stronger recruitment evidence than a study entering an unfamiliar setting.

The need for pilot testing participant recruitment should be judged according to the actual uncertainty.

When Should Informed Consent Procedures Be Prioritized?

Consent procedures deserve particular attention when participants may have difficulty understanding the study, when recruitment involves unequal power relationships, or when procedures are unfamiliar or complex.

For example, university students may be invited to participate in research conducted by their own teachers. The researchers must ensure that participation is voluntary and that students understand any relationship between research participation and academic assessment.

In an online study, researchers may also need to examine whether electronic consent materials are accessible and understandable.

Pilot testing can identify confusing explanations, impractical consent workflows, or misunderstandings about participation.

However, piloting does not replace ethics review or establish that consent is valid merely because participants complete a form.

Procedures involving substantial ethical concerns should be addressed before exposure to unnecessary risk, not deferred until a pilot reveals problems.

When Should Intervention Delivery Be a Priority?

Intervention delivery becomes a major priority when the study evaluates a new, complex, or context-sensitive intervention.

Researchers may need to determine whether staff can deliver the intervention as intended, whether participants can complete the activities, and whether required resources are available.

For example, an AI-supported formative feedback intervention may require teacher training, technical integration, monitoring of feedback quality, and coordination with existing instructional schedules.

Failure in any of these procedures could compromise implementation.

A pilot may examine fidelity, adherence, delivery time, technical problems, and the practical burden on participants or implementers.

Successful delivery, however, does not establish that the intervention produces beneficial outcomes.

Should Outcome Assessment Be Prioritized?

Yes, particularly when the definitive study depends on outcome measures whose administration or completeness is uncertain.

Researchers may need to test whether participants understand assessment instructions, whether measurements can be collected at the intended time points, and whether the resulting records are usable.

For example, a longitudinal online learning study may require students to complete repeated engagement measures over an academic term.

A pilot could reveal that assessment windows conflict with examinations or that participants frequently omit particular questionnaire items.

These findings may support changes to scheduling or administration.

However, successful completion does not automatically establish measurement validity or reliability. Those properties require evidence appropriate to the intended measurement and interpretation.

When Is Participant Retention More Important Than Initial Recruitment?

Retention may deserve greater attention when the study requires repeated participation or long-term follow-up.

A research team may recruit participants successfully but lose a substantial proportion before the primary outcome is measured.

For example, students may enroll in a semester-long study but stop completing follow-up questionnaires during periods of high academic workload.

The pilot can investigate the feasibility of reminders, assessment scheduling, participant communication, and other retention procedures.

Researchers should distinguish retention from outcome data completeness. A participant may remain enrolled but fail to provide a particular measurement.

The priority should reflect which problem would most seriously affect the definitive study.

When Should Randomization Procedures Be Prioritized?

Randomization procedures should be piloted when the proposed trial involves unfamiliar allocation processes, complex eligibility decisions, cluster randomization, or substantial risks of contamination.

For example, a university trial may allocate entire classes rather than individual students to instructional conditions.

The research team may need to examine whether classes can be recruited and allocated according to the protocol, whether allocation concealment is maintained where applicable, and whether participants can be recruited without inappropriate influence from knowledge of allocation.

A pilot can identify operational difficulties, but it should not be used to select allocation procedures based on which arrangement produces the most favorable preliminary treatment effect.

Randomization priorities depend on the definitive trial design.

Should Data Entry and Data Management Be Piloted?

Data management deserves high priority when the study involves complex data flows, multiple collection sites, linked datasets, repeated assessments, or unfamiliar software systems.

For example, a mixed-methods study may require survey records to be linked securely with subsequent interview data.

A pilot can examine whether identifiers function correctly, whether data are transferred accurately, whether access controls are appropriate, and whether missing or duplicate records are detected.

These procedures may be tested using synthetic data before real participant information is collected.

Testing data management can prevent problems that would otherwise become apparent only after substantial data collection.

However, the use of dummy data does not eliminate the need to evaluate procedures under realistic operational conditions where appropriate.

Should the Analysis Pipeline Be Tested?

Testing the analysis pipeline can be useful when the definitive study requires complex data processing, transformations, statistical models, or integration across sources.

Researchers may use synthetic or suitable pilot data to determine whether variables are coded correctly, datasets can be merged, and planned outputs can be produced.

For example, a study using learning analytics may need to transform timestamped event records into defined engagement indicators.

A preliminary test can reveal inconsistent timestamps, missing identifiers, or programming errors.

However, successful execution of an analysis script does not establish that the chosen statistical model is appropriate or that its assumptions will hold in the definitive data.

Technical verification and methodological validation are related but distinct activities.

How Should You Prioritize Procedures That Depend on One Another?

Some research failures arise not from individual procedures but from their interaction.

For example, recruitment may be feasible, and outcome measurement may be technically sound, yet the proposed assessment schedule may be incompatible with participants' availability.

Similarly, intervention delivery may function in a controlled demonstration but become impractical when integrated into ordinary teaching schedules.

A pilot may therefore need to rehearse a sequence of connected procedures rather than test each one independently.

This is an important reason for selecting a pilot rather than a narrower feasibility assessment.

The distinction is explored in When Is a Pilot Study More Appropriate Than a Feasibility Study?.

Should Every Procedure Receive Equal Testing?

No. Equal testing effort may be inefficient when procedures differ substantially in uncertainty and consequences.

For example, a familiar electronic survey platform may require only a technical check, while a newly developed intervention delivery process may need repeated observation.

Nevertheless, low-intensity testing should not mean ignoring essential procedures.

Researchers should distinguish between confirming that a familiar process remains suitable and conducting an intensive evaluation of an unfamiliar process.

The appropriate allocation of effort should be justified by available evidence and the consequences of failure.

Can You Use a Priority Matrix?

Yes. A priority matrix can help organize decisions about which procedures warrant the most attention.

One approach is to classify procedures according to the consequences of failure and uncertainty about feasibility.

This matrix is a practical decision aid, not a validated scoring instrument or a universal methodological standard.

Ethical obligations, critical dependencies, and the possibility of irreversible failure may override the initial classification.

How Do Progression Criteria Affect Prioritization?

Progression criteria specify the conditions under which the definitive study may proceed, require modification, or should not proceed without substantial redesign.

Procedures associated with critical progression criteria often warrant particular attention during the pilot.

For example, if the definitive study cannot achieve adequate statistical power without recruiting a specified number of participants per month, recruitment feasibility becomes a consequential decision criterion.

However, thresholds should be justified by the study's design and resources rather than adopted from arbitrary conventions.

Researchers should consider uncertainty around observed estimates and whether identified problems are remediable.

How Does Pilot Sample Size Affect Prioritization?

The sample size should reflect the feasibility objectives rather than an assumed universal proportion of the definitive sample.

Some procedures can be tested with relatively few observations, particularly when the objective is to identify obvious workflow problems.

Other objectives, such as estimating retention or recruitment proportions with useful precision, may require larger samples.

A small pilot may fail to reveal uncommon problems or variation across institutions.

Researchers should therefore align the sample size with the procedures being evaluated and the decisions the findings must support.

What If the Pilot Reveals an Unexpected Problem?

Unexpected problems should be documented and evaluated according to their consequences.

For example, a pilot focused on recruitment may reveal that participants misunderstand a key questionnaire item.

Even if measurement clarity was not the primary objective, the problem may require attention before the definitive study.

Researchers should distinguish prespecified outcomes from emergent observations and consider whether the new issue changes the study's feasibility.

Additional testing may be warranted when a substantial modification affects procedures that were previously considered adequate.

How Should Pilot Priorities Be Reported?

The protocol should explain why particular procedures were selected for testing and how their feasibility will be assessed.

The final report should identify the procedures actually evaluated, relevant outcomes, limitations, and resulting recommendations.

Researchers should avoid implying that untested procedures were demonstrated to be feasible.

Where a procedure was excluded because existing evidence was considered sufficient, that rationale may be reported when consequential to the study.

The reporting emphasis should remain on what the pilot established about the proposed definitive design.

Prioritizing Pilot Procedures for an Online Learning Intervention

Illustrative Procedure Prioritization Matrix

The following assessments are hypothetical planning judgments, not empirical results or validated risk scores.

The team does not assume that lower-priority procedures can be ignored. Instead, it allocates testing effort according to the uncertainty and potential consequences of each activity.

The pilot also examines interactions among procedures, such as whether teacher training can be completed before intervention delivery and whether assessment schedules are compatible with course activities.

This produces more useful feasibility evidence than testing each component without considering the research workflow.

Common Misconceptions About Prioritizing Pilot Procedures

How to Set Pilot Priorities Before Data Collection

Begin with the definitive study and identify what must work for it to answer the research question.

Then examine which procedures remain uncertain, which failures would be most consequential, and which uncertainties can be reduced through preliminary testing.

After setting priorities, define what evidence will be collected and how it will influence decisions about the definitive study.

The result should be a focused pilot whose procedures correspond to the most consequential uncertainties rather than a collection of tests selected for convenience.

Before Finalizing Pilot Study Procedures, Check:

Frequently Asked Questions About Pilot Study Priorities

Test What Could Most Seriously Undermine the Definitive Study

How to Cite This Guide

This guide is intended to be read, shared, and used in research, teaching, and academic work. If you draw on its ideas, explanations, or other content, please acknowledge the source by citing the guide. Doing so gives appropriate credit and helps your readers locate the original resource.