Can Generative AI Help Design a Pilot or Feasibility Study?

Can AI Help You Test Whether Your Main Study Is Worth Attempting?

You are preparing a research project involving a new AI-supported instructional intervention. Before conducting a full-scale evaluation, you want to determine whether students can be recruited, the intervention can be delivered as planned, and the proposed assessments will work.

You ask generative AI to design a pilot study. It recommends recruiting a small sample, administering a pretest and posttest, and testing whether the intervention produces a statistically significant improvement.

The plan sounds familiar, but does it actually address the purpose of a pilot or feasibility study?

A preliminary study is not necessarily a smaller version of the main study designed to establish effectiveness. Its central purpose may be to resolve uncertainties about whether the intended research can be implemented successfully.

AI Can Help Plan Feasibility Research, but the Study Must Address Feasibility Questions

How AI Can Support Pilot and Feasibility Study Development

What Is the Difference Between a Pilot Study and a Feasibility Study?

The terms are frequently used interchangeably, but methodological guidance distinguishes their purposes.

Eldridge and colleagues (2016) explain that feasibility studies investigate whether a future study can be undertaken and how it should be conducted. Pilot studies are a subset of feasibility studies in which a future study, or part of it, is conducted on a smaller scale.

For example, interviewing instructors about whether an AI tutoring intervention could fit existing teaching arrangements may form part of feasibility research.

Conducting a small-scale version of the proposed randomized trial, including recruitment, allocation, intervention delivery, and outcome assessment, would more specifically resemble a pilot trial.

Researchers should identify the actual purpose rather than rely on the term pilot simply because the sample is small.

Why Should Feasibility Questions Come Before Effectiveness Questions?

A full-scale intervention evaluation may fail because participants cannot be recruited, the intervention is inconsistently delivered, or important outcome data cannot be collected.

These problems may exist independently of whether the intervention has a beneficial effect.

For example, an AI tutoring system might appear promising, but students may not use it consistently, technical access may be unreliable, or instructors may find implementation incompatible with course schedules.

A feasibility study can investigate these uncertainties before substantial resources are committed to a definitive evaluation.

Its primary questions should therefore concern whether the future research can be conducted and what changes are necessary.

What Feasibility Outcomes Can AI Help Identify?

AI may help organize feasibility objectives across recruitment, implementation, measurement, and participant experience.

Not every study requires all these outcomes. The appropriate selection depends on the uncertainties that could prevent the main study from proceeding.

What Are Progression Criteria?

Progression criteria are prespecified conditions used to inform whether a study should proceed to a larger investigation, proceed with modifications, or undergo substantial reconsideration.

Avery and colleagues (2017) discuss the importance of progression criteria in external pilot randomized trials.

For example, a researcher might specify minimum recruitment, retention, and intervention-delivery targets before beginning the pilot.

These criteria should reflect the requirements of the intended main study, relevant prior evidence, and stakeholder or methodological judgment.

AI may suggest candidate thresholds, but it cannot establish their appropriateness without sufficient contextual information.

Should Progression Criteria Use a Traffic-Light System?

Some pilot studies use a traffic-light approach to organize progression decisions.

The labels themselves do not determine scientific decisions. Thresholds and their interpretation should be established before the results are examined, while allowing justified consideration of qualitative evidence and contextual circumstances.

A red result does not necessarily mean that the intervention is ineffective. It may indicate that the research procedures or implementation arrangements are not sufficiently feasible.

Can AI Recommend a Pilot Study Sample Size?

AI can explain sample-size considerations, but it should not provide a universal number without examining the study's objectives.

For example, a pilot designed to estimate a recruitment proportion with a specified degree of precision may require a different sample size from one intended to examine intervention delivery across several classrooms.

Whitehead and colleagues (2016) discuss sample-size considerations for pilot randomized trials in relation to planning future studies.

However, their recommendations apply to particular objectives and assumptions. They should not be treated as a universal rule for every pilot or feasibility investigation.

Researchers should determine which feasibility parameter needs estimation, the required precision, the unit of recruitment or randomization, and the resources available.

Should a Pilot Study Test Whether the Intervention Is Effective?

A pilot study may collect clinical, educational, or behavioral outcomes to assess whether measurement procedures work and to obtain preliminary information relevant to planning.

However, small pilot studies are generally not designed to provide definitive tests of intervention effectiveness.

Using statistical significance as the main criterion for deciding whether to proceed can be misleading because pilot samples are often too small to estimate intervention effects precisely.

Eldridge and colleagues (2016) emphasize the feasibility purpose of pilot studies conducted in preparation for randomized trials.

Researchers should distinguish feasibility outcomes from exploratory effectiveness outcomes and avoid interpreting an inconclusive significance test as evidence that the intervention does not work.

Can AI Help Design a Pilot Randomized Trial?

Yes. AI may assist in planning a smaller-scale evaluation of recruitment, randomization, allocation concealment, intervention delivery, follow-up, and data collection.

For example, a researcher might pilot a proposed randomized study of AI-generated feedback to determine whether students can be assigned to conditions and whether blinded writing assessment can be implemented.

However, a pilot trial should preserve the essential procedures that need testing. Omitting randomization from a pilot intended to evaluate the feasibility of randomization would leave that particular uncertainty unresolved.

Researchers should examine randomization procedures and feasible blinding arrangements when these form part of the intended definitive trial.

Can Feasibility Research Use Qualitative Methods?

Yes. Interviews, observations, and other qualitative approaches may provide important evidence about why particular procedures are acceptable, difficult, or impractical.

For example, a recruitment target may not be achieved because students misunderstand the study, instructors are reluctant to release class time, or institutional procedures delay enrollment.

Numerical recruitment rates identify the problem but may not explain its causes.

Qualitative inquiry can help researchers understand implementation barriers and develop justified modifications.

Where quantitative and qualitative components are intentionally integrated, the feasibility study may incorporate a mixed-methods approach. The rationale for integration should be explicit.

How Can AI Help Identify Feasibility Problems Specific to Educational Technology?

Educational technology interventions may involve requirements that are not adequately captured by generic research templates.

These may include platform access, internet reliability, device compatibility, institutional data protection requirements, licensing restrictions, technical support, and the ability to maintain consistent intervention exposure.

For example, a study using a commercial generative AI platform may be affected by changes to model behavior, access conditions, or available features during implementation.

AI may help identify these risks, but the researcher must verify the actual platform conditions and institutional requirements.

This is one reason to examine whether AI recommendations overlook field-specific constraints.

What Should a Feasibility Study Produce?

A useful feasibility study should inform a practical decision about the future investigation.

Its outputs may include recruitment estimates, retention information, implementation findings, measurement problems, acceptability evidence, and recommended protocol modifications.

It should also document which uncertainties remain unresolved.

A favorable feasibility result does not establish intervention effectiveness. Conversely, identifying implementation problems may be a valuable outcome because it prevents an inadequately prepared definitive study from proceeding.

How Can You Use AI to Plan a Feasibility Study Systematically?

The proposed feasibility study should then undergo appropriate methodological and ethical review before implementation.

Designing a Feasibility Study for an AI Tutoring Intervention

The researcher works with methodological advisers and institutional stakeholders to establish progression criteria appropriate to the planned main trial.

Suppose the feasibility study finds that recruitment is adequate but many students cannot access the tutoring system reliably.

The decision: The researcher revises technical support and intervention delivery procedures before considering a larger trial.

The pilot's value lies in identifying an implementation problem that could compromise the definitive evaluation.

Even if preliminary achievement differences favor the AI tutoring group, the researcher avoids treating those results as definitive evidence of effectiveness because the pilot was not designed or powered for that conclusion.

Common Mistakes in AI-Generated Pilot and Feasibility Study Plans

Ask AI What Must Be Learned Before the Main Study Can Proceed

When planning a pilot or feasibility study, begin by identifying the uncertainties that could undermine the proposed definitive investigation.

AI can then help translate those uncertainties into feasibility objectives, indicators, data collection procedures, and progression decisions.

A Prompt for Designing a Feasibility Study

Researchers should then examine whether the proposed preliminary study is itself feasible to implement and whether the design addresses the uncertainties that genuinely matter.

Where the intended main study involves experimental procedures, the pilot should also consider validity threats that may emerge during recruitment, intervention delivery, and outcome assessment.

A useful feasibility study should reduce uncertainty about the next research decision rather than merely produce a preliminary p-value.

Before Finalizing an AI-Assisted Pilot or Feasibility Study

Frequently Asked Questions About AI and Pilot or Feasibility Studies

A Good Pilot Tells You Whether the Main Study Can Be Done, Not Whether You Have Already Won

How to Cite This Guide

This guide is intended to be read, shared, and used in research, teaching, and academic work. If you draw on its ideas, explanations, or other content, please acknowledge the source by citing the guide. Doing so gives appropriate credit and helps your readers locate the original resource.