Should You Preregister Planned Subgroup Analyses?

Should You Decide Which Subgroups to Compare Before Seeing the Results?

Suppose an educational intervention produces a modest overall improvement in achievement. When you examine the results separately, the intervention appears more effective among students with low baseline achievement than among students with high baseline achievement. Does that mean the intervention genuinely works better for one group?

Possibly, but subgroup findings require careful interpretation. Differences may reflect real variation in intervention effects, sampling uncertainty, differences in baseline risk, or the consequences of examining many possible comparisons.

Preregistering planned subgroup analyses can clarify which differences researchers intended to investigate before observing the results. The challenge is specifying these analyses sufficiently while retaining the ability to investigate unexpected patterns.

Yes, When Subgroup Comparisons Will Support Confirmatory Claims

How to Preregister Subgroup Analyses That Produce Interpretable Evidence

What Is a Subgroup Analysis?

A subgroup analysis examines whether an outcome, association, or intervention effect differs across categories of a population. Subgroups might be defined by age, baseline achievement, prior experience, educational setting, or other theoretically relevant characteristics.

In intervention research, the central question is often whether the effect of treatment varies across groups. This is known as effect modification or treatment effect heterogeneity.

For example, researchers might hypothesize that an instructional intervention produces a larger improvement among students with limited prior knowledge than among students with extensive prior knowledge.

Not every comparison conducted within a subgroup evaluates effect modification. Reporting a treatment effect separately for two groups is descriptive unless the analysis also addresses whether those effects differ.

Why Is Preregistration Particularly Important for Subgroup Analyses?

Researchers can divide a sample in numerous ways. A study may contain information about age, academic program, achievement level, gender, prior experience, learning preferences, and institutional setting.

Each variable may support several subgroup definitions, thresholds, or combinations. When researchers examine many alternatives and emphasize only statistically favorable findings, the probability of misleading patterns increases.

This concern is especially important when subgroup hypotheses are developed after observing the same data used to evaluate them.

Preregistration establishes which comparisons were planned and reduces ambiguity about whether a finding represents a prospective hypothesis test or an exploratory discovery. It is one of the consequential design and analysis decisions worth specifying in advance.

Which Elements of a Subgroup Analysis Should Be Prespecified?

A statement such as "we will examine whether effects differ by student characteristics" leaves substantial analytical discretion.

A more complete specification should address the following elements.

The required detail depends on the research design. Nevertheless, another analyst should be able to determine which subgroup comparison was intended without choosing definitions after inspecting the results.

Why Should Subgroup Variables Usually Be Measured Before the Intervention?

For randomized intervention studies, subgroup analyses based on baseline characteristics are generally easier to interpret than those based on characteristics measured after assignment.

A baseline variable cannot be caused by the subsequently assigned intervention. By contrast, a post-treatment variable such as attendance, engagement, or adherence may itself be affected by treatment.

Conditioning on such a variable can introduce selection bias or change the causal question being addressed.

For example, comparing treatment effects only among students who completed every learning activity may not preserve the comparability established through randomization. Completion may depend on motivation, access, treatment experience, or other characteristics associated with outcomes.

Post-treatment subgroup questions are not inherently illegitimate, but they may require specialized causal methods and assumptions rather than ordinary subgroup comparisons.

Should You Categorize a Continuous Variable to Create Subgroups?

Not automatically. Dividing a continuous characteristic such as age, achievement, or baseline proficiency into high and low categories may simplify presentation, but it can discard information and make results sensitive to the selected cutoff.

If a threshold has a substantive or established methodological rationale, categorical subgroups may be appropriate. Otherwise, modeling the characteristic continuously may be preferable.

For example, researchers could examine whether the intervention effect changes as baseline achievement increases rather than dividing students at the sample median.

Nonlinear relationships may require flexible modeling, such as appropriately specified spline functions. Such choices should be justified and, where they support confirmatory claims, prespecified.

Choosing the cutoff that produces the strongest interaction after examining the data introduces additional selection concerns.

Why Is an Interaction Test Usually More Appropriate Than Comparing Separate p-Values?

A common mistake is concluding that two subgroups differ because the treatment effect is statistically significant in one subgroup but not in another.

That conclusion does not follow. Statistical significance depends on effect size, uncertainty, and sample size. Two subgroups can have similar estimated effects while producing different p-values.

When the research question concerns whether effects differ, an appropriate interaction test or direct contrast between subgroup-specific effects is generally required.

This example uses a linear model with an additive interaction. Interactions in logistic, survival, and other nonlinear models require interpretation on the appropriate scale. An interaction on one scale does not necessarily imply the same pattern on another.

How Do Multiple Subgroup Tests Affect Interpretation?

Examining many subgroup hypotheses creates a multiple-testing problem. Even when no genuine effect modification exists, some comparisons may appear statistically significant by chance.

Researchers should identify the intended family of confirmatory tests and determine whether an adjustment or structured testing procedure is appropriate.

Possible strategies include limiting the number of confirmatory subgroup hypotheses, using hierarchical testing, controlling a relevant error rate, or applying appropriate multilevel or shrinkage models.

No single method is universally best. The choice depends on whether the analyses are confirmatory or exploratory, the intended claims, and the dependence among tests.

Simply declaring every subgroup analysis statistically significant at 0.05 without considering multiplicity may overstate the evidence.

Do Subgroup Analyses Require Larger Samples?

Often, particularly when the objective is to detect differences between subgroup-specific effects.

A study powered to estimate an overall treatment effect may have insufficient power to detect interactions. Subgroups may contain relatively few participants, and the uncertainty around the difference between effects can be substantial.

Failure to detect an interaction therefore does not necessarily demonstrate that effects are identical across groups.

If a subgroup hypothesis is central to the study, its expected precision or power should inform the sample-size justification and stopping plan.

When subgroup analyses are secondary, researchers should communicate their uncertainty and avoid interpreting imprecise estimates as definitive evidence of either heterogeneity or equivalence.

Can You Add Subgroup Analyses After Seeing the Results?

Yes. Unexpected subgroup patterns can generate valuable research questions.

However, analyses developed after inspecting outcomes should be reported as exploratory or post hoc. A compelling explanation written after observing a pattern does not establish that the hypothesis was prospective.

Researchers should disclose how the subgroup was identified, how many related comparisons were considered when feasible, and whether the result is robust to reasonable alternative specifications.

Independent replication or prospective evaluation in a new dataset may provide stronger evidence than repeated examination of the same observations.

The principles for changing preregistered decisions also apply when an originally planned subgroup analysis must be revised.

Are Subgroup Analyses Appropriate for Observational Research?

Yes, but interpretation requires particular care.

In observational studies, differences across groups may reflect effect modification, confounding, selection processes, measurement differences, or other sources of heterogeneity.

Researchers should prespecify the relevant subgroup definitions, adjustment strategy, interaction model, and assumptions supporting the intended interpretation.

A statistically significant interaction in an observational analysis does not, by itself, establish a causal difference in effects. The strength of the claim depends on the underlying design and identification assumptions.

Testing Whether Digital Feedback Works Differently Across Achievement Levels

Suppose the estimated intervention benefit is four points at a relatively low baseline score and two points at a relatively high baseline score. Those estimates alone do not establish effect modification. The relevant evidence concerns the estimated difference between effects, its uncertainty, and the validity of the prespecified model.

The study may also reveal unexpected patterns involving academic program or prior technology use. Such findings can be reported as exploratory without being retrospectively presented as planned subgroup hypotheses.

Common Mistakes in Subgroup Analysis and Preregistration

How to Decide Which Subgroup Analyses Deserve Preregistration

Start with the substantive question. A subgroup analysis is most useful when there is a credible reason to expect meaningful heterogeneity and the resulting information could influence scientific understanding or practical decisions.

A large number of loosely motivated subgroup comparisons may create more interpretive problems than useful evidence.

Subgroup planning should also align with the prespecified primary outcome. Using different outcomes for different subgroups without a justified plan can make interpretation particularly difficult.

When subgroup comparisons are underpowered, prioritize effect estimates and uncertainty rather than treating statistical significance as a simple indicator of whether a difference exists.

Before Preregistering Subgroup Analyses

Frequently Asked Questions About Preregistering Subgroup Analyses

Preregister Subgroup Hypotheses Before Searching for Differences

How to Cite This Guide

This guide is intended to be read, shared, and used in research, teaching, and academic work. If you draw on its ideas, explanations, or other content, please acknowledge the source by citing the guide. Doing so gives appropriate credit and helps your readers locate the original resource.