How Do You Design a Study So Others Can Assess Whether Its Findings Apply to Different Settings?

Could Another Researcher Determine Whether Your Findings Apply to Their Institution?

You complete a study showing that an AI-supported learning platform improves students' examination performance. The methods describe the research design, sample size, statistical analysis, and estimated treatment effect.

A researcher at another university wants to implement the platform. However, your article provides little information about students' digital proficiency, institutional infrastructure, instructor support, or how the intervention was actually delivered.

The researcher cannot determine whether the original conditions resemble those at their institution. Even if the causal estimate is credible, its relevance elsewhere remains difficult to assess.

Applicability is not something readers can evaluate from an effect estimate alone. Researchers must design studies that collect and report enough information about participants, settings, interventions, and relevant mechanisms for others to judge whether findings might extend beyond the original context.

How Can You Design Research So Others Can Assess Its Applicability?

Building Applicability Into Research Design, Data Collection, and Reporting

Why Must Applicability Be Considered Before Data Collection?

Researchers sometimes discuss external validity only after completing their analysis. By then, important contextual information may be unavailable.

For example, a study may establish that an instructional intervention improved performance but fail to record whether students received additional technical assistance or how frequently instructors provided guidance.

Those conditions may influence intervention effectiveness and therefore matter to institutions considering adoption.

If they were not documented, subsequent readers may be unable to determine whether the intervention can be implemented under comparable circumstances.

Planning for applicability before data collection allows researchers to identify relevant contextual information, select informative settings, and establish reporting procedures.

The objective is not to predict every future application. It is to preserve the information reasonably necessary for assessing the intended uses of the findings.

Begin by Defining the Target Population and Intended Application

Researchers should identify who the findings are intended to inform and under what circumstances they might be used.

For example, a digital learning intervention might be intended for first-year engineering students in universities with blended learning programs.

That target is more informative than a vague claim about students generally.

A clear population definition helps determine which participant characteristics, institutional conditions, and implementation factors should be measured.

It also helps distinguish direct evidence from broader extrapolation.

Researchers should therefore define the population to which findings are intended to apply before selecting sites and establishing recruitment procedures.

What Information About Participants Should Be Collected?

Participant information should describe characteristics relevant to the research question, outcome, and potential applicability.

For educational technology research, these may include academic level, prior achievement, digital proficiency, device access, and relevant learning experiences.

However, collecting every available demographic variable is neither necessary nor automatically informative.

Researchers should prioritize characteristics that define eligibility, influence participation, predict outcomes, or plausibly modify intervention effects.

Where personal information is collected, the study should follow applicable ethical and data protection requirements.

Reporting may use appropriate aggregated descriptions rather than identifiable participant-level information.

The purpose is to help readers understand who was studied and how those participants relate to the intended population.

Why Must Eligibility and Recruitment Procedures Be Transparent?

Readers need to know not only who participated but also who could have participated and how the final sample was formed.

Eligibility criteria determine the population permitted to enter the study. Recruitment procedures determine which eligible individuals were approached and enrolled.

For example, requiring personal laptop ownership may exclude students whose access depends on shared devices.

Recruiting exclusively through voluntary online announcements may further favor students who are digitally engaged and frequently connected.

These processes can produce a sample whose characteristics differ from the intended users.

Researchers should document inclusion and exclusion criteria, recruitment channels, participation patterns, and important reasons for nonparticipation where available.

This information helps readers evaluate how eligibility restrictions affect real-world applicability.

What Information About Research Settings Matters?

Contextual information should reflect conditions that may influence the phenomenon or intervention being studied.

For educational technology interventions, relevant features may include technological infrastructure, instructional arrangements, class size, faculty preparation, technical support, and institutional policies.

However, the appropriate contextual variables depend on the intervention.

A study of automated writing feedback may require information about assessment practices and instructor involvement. A study of virtual laboratories may require information about hardware, software, and practical learning requirements.

Researchers should explain why particular contextual characteristics matter rather than presenting an extensive but uninformative institutional profile.

The goal is to identify conditions that help readers judge whether the intervention and its effects are likely to be relevant elsewhere.

How Can Researchers Identify Contextual Factors Worth Measuring?

Begin with a plausible explanation of how the intervention is expected to influence outcomes.

Suppose an AI-supported learning platform improves performance by providing timely feedback and opportunities for repeated practice.

Its effectiveness may depend on students having sufficient access to the platform, understanding the feedback, and receiving instructional guidance when needed.

These mechanisms suggest relevant contextual measures such as connectivity, engagement opportunities, and facilitator support.

Researchers may consult prior literature, pilot studies, implementation evidence, and stakeholder knowledge to refine the selection.

Contextual measurement should follow substantive reasoning rather than an assumption that every observable institutional difference must be recorded.

Why Is Intervention Description Essential?

Readers cannot assess applicability if they do not know what intervention was actually evaluated.

A statement that students used an AI-supported platform is insufficient to establish its instructional components.

Researchers should describe the intervention's purpose, materials, procedures, delivery schedule, personnel, participant activities, and relevant adaptations.

For digital interventions, additional details may include platform functionality, feedback mechanisms, access arrangements, and the role of instructors.

The Template for Intervention Description and Replication, known as TIDieR, provides a structured framework for describing interventions sufficiently for replication and interpretation.

Where intervention components are proprietary or cannot be fully disclosed, researchers should still provide as much scientifically relevant information as permitted and acknowledge limitations on reproducibility.

How Is Intervention Fidelity Different From Applicability?

Intervention fidelity concerns whether the intervention was delivered as intended.

Applicability concerns whether findings can inform another population or setting.

These concepts are related because the effect observed in one study may depend on how the intervention was implemented.

For example, a platform evaluated with intensive instructor guidance may not produce the same effect when deployed without that guidance.

Researchers should distinguish essential intervention components from flexible implementation arrangements.

This distinction helps readers determine whether adapting the intervention to their setting preserves the treatment strategy or creates a substantively different intervention.

Should Every Research Site Use Identical Procedures?

Not necessarily. The appropriate degree of standardization depends on the research question.

Explanatory studies may prioritize consistent intervention delivery to investigate effects under specified conditions.

Pragmatic studies may allow flexibility consistent with ordinary practice.

However, researchers should specify which components must remain consistent and which may be adapted.

For example, instructors may use different classroom examples while retaining the same essential feedback and practice procedures.

When adaptations are permitted, their nature and rationale should be documented.

Without this information, readers may struggle to determine whether differences across sites reflect context, implementation, or fundamentally different interventions.

How Can Site Selection Improve Applicability?

Site selection can provide evidence about how findings operate under different contextual conditions.

For example, recruiting universities with varying levels of technological infrastructure may help researchers examine whether connectivity influences intervention delivery.

However, multiple sites do not automatically establish population representativeness.

The value of multisite research for generalizability depends on how settings were selected and what relevant variation they provide.

Researchers should also distinguish purposive contextual diversity from probability-based site sampling.

Both can support useful research, but they justify different forms of inference.

What Is the Role of Effect Modifiers?

An effect modifier is a characteristic across which the causal effect of an intervention differs on a specified effect scale.

For example, a learning platform may benefit students differently depending on prior digital proficiency or access to technical support.

If such characteristics are measured, researchers may investigate whether the estimated intervention effect varies across relevant subgroups.

However, subgroup comparisons require careful design and analysis. Interaction estimates may be imprecise, and a nonsignificant interaction does not establish identical effects.

Researchers should prespecify important effect-modification hypotheses where possible and avoid treating exploratory subgroup findings as definitive.

Measuring plausible effect modifiers may also support future transportability analyses across populations.

What Should Be Reported About the Comparison Condition?

The comparison condition is essential to interpreting the estimated effect.

For example, conventional instruction may mean very different things across universities.

At one institution, students may receive extensive tutoring and digital resources. At another, conventional instruction may involve limited supplementary support.

An intervention effect is always relative to a specified comparison condition.

Researchers should therefore describe the comparator's instructional activities, resources, personnel, and relevant co-interventions.

Without this information, another institution may incorrectly assume that the reported effect compares the intervention with its own existing practices.

Why Must Outcome Measurement Be Described?

Outcome definitions and measurement procedures affect whether findings can be compared across settings.

Researchers should report what was measured, which instruments were used, how assessments were administered, and how scores were calculated.

For example, an intervention effect on a locally developed examination may not translate directly into an effect on a standardized assessment measuring a different range of competencies.

Even when two studies use similar outcome labels, their measures may differ in content, difficulty, or interpretation.

Researchers should establish appropriate measurement comparability rather than assume that identical terminology guarantees equivalent outcomes.

Documentation of measurement procedures and their consistency supports both internal validity and assessment of applicability.

Can Implementation Data Help Explain Why Findings Differ Across Settings?

Yes. Implementation data can help distinguish whether different outcomes reflect contextual characteristics, intervention delivery, participant engagement, or other factors.

For example, two universities may adopt the same platform but differ in how frequently students complete adaptive exercises.

Researchers may collect platform activity records, facilitator logs, observations, or participant feedback to characterize implementation.

However, implementation measures should not be interpreted as causal explanations automatically.

Engagement may be influenced by student motivation, prior achievement, or other characteristics that also affect outcomes.

Analyses linking engagement to outcomes require attention to confounding and postassignment selection.

Implementation information is valuable for interpretation even when it does not identify a separate causal effect.

How Does Pragmatic Research Support Applicability?

Pragmatic research evaluates interventions under conditions intended to inform routine decisions.

It may involve broader eligibility, ordinary service settings, flexible intervention delivery, and outcomes relevant to practice.

The PRECIS-2 framework helps researchers examine whether trial design choices align with a pragmatic or explanatory purpose.

However, pragmatic design does not automatically guarantee external validity.

A pragmatic trial conducted in a narrow set of institutions may still provide limited evidence about other environments.

Researchers should distinguish realism of implementation from representation of the target population.

What Reporting Frameworks Can Support Applicability?

Reporting guidelines can help researchers identify information needed for transparent interpretation.

Researchers should select reporting frameworks appropriate to their study design rather than treating one checklist as sufficient for every methodological tradition.

Reporting guidelines are useful for transparency, but the credibility of applicability claims depends on the underlying evidence and assumptions.

How Can Qualitative Research Support Transferability?

Qualitative studies may support transferability by providing detailed information about participants, contexts, processes, and the circumstances under which interpretations were developed.

For example, a study of instructors' experiences with AI-supported assessment may describe institutional policy, teaching responsibilities, professional preparation, and implementation challenges.

Readers can then consider whether those circumstances resemble their own settings.

However, contextual description does not prove that findings will transfer.

Researchers should distinguish qualitative transferability from statistical population inference and avoid imposing quantitative representativeness criteria indiscriminately across methodological traditions.

Should Researchers Make Their Materials and Protocols Available?

Where ethically, legally, and practically appropriate, sharing research protocols, intervention materials, measurement instruments, and analytical documentation can help others assess and reproduce the study.

For educational technology interventions, useful documentation may include instructional sequences, platform functionality, assessment procedures, and implementation guidance.

However, intellectual property restrictions, participant confidentiality, institutional agreements, or security concerns may limit what can be shared publicly.

Researchers should distinguish materials that can be openly released from information that can only be described or accessed under controlled conditions.

Transparency should be maximized within legitimate ethical and legal constraints, not pursued at the expense of participant protection.

Can Researchers Prove Applicability Through Detailed Reporting Alone?

No. Detailed reporting makes applicability easier to assess, but it does not establish that findings will hold in another population.

For example, researchers may thoroughly describe a digital learning intervention and its implementation conditions while lacking evidence about its effect among students with limited connectivity.

Readers can recognize the difference, but the report does not resolve it.

Formal generalization may require target-population information, measured effect modifiers, adequate overlap, and defensible assumptions.

In other circumstances, replication or new intervention studies may be necessary.

The distinction is between making a claim assessable and establishing that the claim is true.

How Should Applicability Limitations Be Written?

Applicability limitations should identify specific differences between the study and intended application rather than relying on generic statements.

For example, saying that all participants had reliable internet access and that intervention performance under intermittent connectivity was not evaluated is more informative than stating that the sample may not be generalizable.

Similarly, describing the level of facilitator support helps readers determine whether the intervention resembles what their institution could implement.

Researchers should distinguish demonstrated limitations from plausible concerns and identify additional evidence needed for broader conclusions.

A useful limitations section clarifies the boundary of the evidence rather than merely acknowledging that all studies have limitations.

Designing an Educational Technology Study for Future Institutional Adoption

The team develops a prospective applicability documentation plan.

Suppose the intervention produces favorable outcomes across the participating institutions. The researchers may have stronger evidence about several observed contexts than they would have obtained from a single site.

However, they should not conclude that the effect will be identical at every university.

Instead, the documented contextual information allows other institutions to assess whether their circumstances resemble those studied and what additional evidence or adaptation may be necessary.

Common Misconceptions About Designing Research for Applicability

How to Build an Applicability Assessment Into Your Research Protocol

Before recruitment, identify who might use the findings and what information they would need to determine whether the results are relevant to their setting.

Then specify the participant, institutional, intervention, and measurement characteristics most likely to influence that judgment.

Collect and report these characteristics systematically without treating contextual description as proof of generalizability.

Use reporting guidance appropriate to the study design, but treat those guidelines as support for transparent communication rather than substitutes for methodological reasoning.

When important contextual information cannot be collected, identify the resulting limitation explicitly.

The practical objective is to make the study sufficiently transparent that others can distinguish conditions represented by the evidence from conditions requiring additional assumptions or research.

Applicability Documentation Checklist for Research Design and Reporting

Frequently Asked Questions About Research Applicability

Make Applicability Assessable Before Asking Others to Generalize

How to Cite This Guide

This guide is intended to be read, shared, and used in research, teaching, and academic work. If you draw on its ideas, explanations, or other content, please acknowledge the source by citing the guide. Doing so gives appropriate credit and helps your readers locate the original resource.