Should You Preregister Data Exclusion Rules Before Analysis?

Should You Decide Which Data to Exclude Before Examining the Results?

You have completed data collection and are preparing your dataset for analysis. Some participants have incomplete responses, several observations appear unusually high, and a few records contain values that seem impossible. Should you exclude these observations? More importantly, should the rules for making those decisions have been established before you examined the data?

Data exclusions can be necessary for valid analysis. However, they can also change estimated effects, statistical significance, and the population represented by the findings. When exclusion decisions are made after researchers know how those decisions affect their results, it becomes difficult to distinguish legitimate data-quality procedures from selective analysis.

Preregistering exclusion rules can reduce this ambiguity, but it requires careful distinctions between invalid data, unusual yet valid observations, missing measurements, and participants who do not follow the intended research procedures.

Yes, Prespecify Consequential Data Exclusion Decisions

How to Prespecify Data Exclusion Rules Without Introducing Bias

What Is a Data Exclusion Rule?

A data exclusion rule specifies when a participant, observation, measurement, or record will be omitted from a particular analysis.

Such rules may address duplicate records, confirmed measurement errors, invalid responses, technical failures, implausible values, or other circumstances that prevent data from being interpreted as intended.

Importantly, exclusion is not necessarily an all-or-nothing decision. A participant may contribute valid baseline information but have an invalid measurement at one follow-up occasion. Removing that measurement does not automatically justify removing the participant's entire record.

The appropriate unit of exclusion should therefore be specified.

Preregistration should explain which type of exclusion applies and why the resulting analysis remains appropriate for the research question.

Why Should Data Exclusion Rules Be Established in Advance?

Researchers often have several defensible ways to prepare the same dataset. They may choose different outlier thresholds, apply alternative missing-data procedures, remove incomplete records, or define invalid responses differently.

These choices can produce different estimates and conclusions. If researchers examine the results of several exclusion strategies and report only the most favorable one, the apparent strength of the evidence may be misleading.

Prespecification establishes which decisions were intended before relevant results became available. It helps distinguish planned data preparation from outcome-informed changes and makes the analysis easier to reproduce.

Nevertheless, preregistration does not establish that an exclusion rule is scientifically appropriate. A poorly justified rule can remain biased even when it was specified in advance.

Which Types of Data Exclusions Should Be Preregistered?

Prioritize rules that could change the composition of the analysis population, the estimated relationship, or the uncertainty associated with the findings.

Not every study needs every category. The rules should reflect foreseeable problems in the actual measurement and data collection procedures.

What Is the Difference Between an Invalid Observation and an Outlier?

An invalid observation contains information that cannot reasonably be treated as a valid measurement under the intended procedure. An outlier is an observation that differs substantially from other observations according to a statistical or substantive criterion.

These categories are not equivalent.

Suppose a questionnaire records age as 450 years. If the study concerns contemporary university students, that value is almost certainly a recording or entry error. It may require verification, correction, or treatment as missing.

By contrast, a participant who scores much higher than classmates on a valid achievement assessment may simply be an unusually high-performing student. Excluding that score because it is extreme could remove genuine variation from the population.

Outliers may be scientifically informative. They can reveal meaningful heterogeneity, model misspecification, or unexpected processes. Researchers should investigate their origin rather than assume that removing them improves the analysis.

Should You Preregister a Statistical Outlier Threshold?

When an outlier rule is consequential to the primary analysis, advance specification is generally advisable. However, the threshold should have a methodological justification.

Common approaches include standardized scores, interquartile-range fences, model residual diagnostics, and domain-specific plausibility limits. These methods identify different properties and are not interchangeable.

For example, an interquartile-range rule may identify observations below the first quartile minus 1.5 times the interquartile range or above the third quartile plus 1.5 times that range.

A flagged observation is not necessarily erroneous or unsuitable for analysis. The calculation identifies values outside conventional fences; it does not demonstrate that those values should be deleted.

For skewed distributions, bounded scales, clustered data, or other complex structures, the conventional rule may flag many valid observations. Researchers should choose diagnostics that reflect the measurement scale and analysis model.

Prespecification should distinguish the rule for identifying observations requiring investigation from the rule, if any, authorizing exclusion.

Should You Exclude Participants Who Fail Attention Checks?

Attention checks are sometimes used in surveys and experiments to identify responses that may not reflect adequate engagement with the task.

However, failing an attention check does not always establish that the participant's remaining responses are invalid. A failure may result from ambiguous instructions, language difficulties, accessibility barriers, or an isolated mistake.

If attention checks will influence the primary analysis, specify their content or operational criteria, the number of failures required for exclusion, and the intended treatment of affected records.

Researchers should also consider whether the checks are appropriate for the population and whether exclusion rates differ systematically across experimental conditions or participant groups.

Where feasible, analyses retaining all otherwise eligible observations can help assess whether conclusions depend on the exclusion rule.

How Should Missing Data Be Handled?

Missing data should not automatically trigger exclusion of an entire participant.

Complete-case analysis, which uses only records containing all variables required for a particular analysis, can be appropriate under some conditions. However, it may reduce precision and introduce bias when the probability of having complete data is related to relevant variables or outcomes.

Alternative approaches include multiple imputation, likelihood-based estimation, and models designed for particular missing-data structures. Their validity depends on assumptions about the missingness process and the statistical model.

A preregistration should identify the intended strategy and any relevant conditional procedures. It should also specify how missingness will be described and whether sensitivity analyses will examine departures from key assumptions.

The primary outcome definition should be coordinated with these decisions, particularly when incomplete assessments affect scoring or analysis eligibility.

Why Are Post-Randomization Exclusions Especially Sensitive?

Random assignment is intended to create comparable groups with respect to potential outcomes, subject to chance variation. Excluding participants after assignment can compromise that comparability when exclusion depends on treatment, adherence, outcomes, or related characteristics.

For example, excluding students who did not complete an intervention may create groups that differ in motivation or academic preparation, even if the original assignment was randomized.

The appropriate analysis depends on the intended estimand and design. An assignment-based analysis and a per-protocol analysis may address different causal questions and require different assumptions.

Researchers should therefore distinguish participant eligibility criteria from post-enrollment analysis exclusions. A participant who was eligible at enrollment does not automatically become ineligible because of nonadherence or missing follow-up information.

Can You Use Robust Statistical Methods Instead of Deleting Outliers?

Sometimes. Robust estimation, suitable transformations, distributionally appropriate models, and sensitivity analyses may reduce the influence of extreme observations without discarding valid data.

For example, an analysis of highly skewed response times may benefit from a model appropriate for positive, asymmetric measurements rather than automatic deletion of the slowest responses.

However, robust methods are not universally preferable. They estimate particular quantities under particular assumptions and may answer a different question from the originally planned analysis.

When several defensible methods are available, identify the primary approach and the role of alternatives in the prespecified analysis plan.

What Should Happen When an Unexpected Data Problem Appears?

No preregistration can anticipate every software error, measurement failure, or data-integrity problem.

If an unforeseen issue arises, first investigate its origin and determine whether the affected information can be corrected or validated. Exclusion should not be the automatic first response.

Where a change is necessary, document the original rule, the newly identified problem, the revised procedure, the decision date, and the information available when the decision was made.

Ideally, the decision should be made without knowledge of its effect on the principal results. Blinded data review or consultation with an independent analyst may be useful when feasible.

Follow the principles for amending a preregistered research plan. Transparency about a deviation does not automatically eliminate its statistical consequences, but it allows readers to evaluate those consequences.

How Should Exclusions Be Reported in the Final Study?

The final report should distinguish the number of observations initially obtained, the number excluded, the reasons for exclusion, and the number included in each important analysis.

Where applicable, report whether exclusions were prespecified, whether the criteria changed, and how conclusions differ when alternative defensible rules are applied.

For randomized trials, the CONSORT reporting framework emphasizes transparent participant flow and analysis populations. For observational research, STROBE addresses participant selection, missing data, and relevant reporting considerations.

Researchers should follow reporting guidance appropriate to their design rather than assuming that a single exclusion table satisfies every methodological requirement.

Preregistering Data Exclusions in an Online Learning Experiment

Suppose 200 students are randomized, 190 provide posttest records, and three of those records contain unrecoverable technical errors. The researchers would report these quantities separately rather than simply describing the final dataset as containing 187 valid participants.

Whether the primary analysis includes additional information from participants with missing posttests depends on the specified estimand and statistical method. The example illustrates transparent data handling, not a universal recommendation to conduct complete-case analysis.

Common Misconceptions About Excluding Research Data

How to Decide Whether Data Should Be Excluded

Begin by asking whether the observation is valid for the intended measurement and analysis. Do not begin by asking whether its inclusion strengthens or weakens the hypothesis.

Next, determine whether the problem can be corrected, whether only part of the record is affected, and whether exclusion would introduce selection bias or change the quantity being estimated.

When exclusions materially affect the results, consider reporting analyses with and without the affected observations, provided both analyses are scientifically interpretable. Such comparisons can reveal sensitivity to the exclusion rule, although they do not automatically identify which estimate is less biased.

Coordinate the exclusion plan with sample-size and stopping decisions. Anticipated exclusions and missing observations can affect statistical precision, recruitment planning, and the interpretation of the achieved sample.

Above all, maintain an auditable record. Researchers should be able to explain what was removed, why it was removed, when the decision was made, and how the resulting analysis relates to the original research question.

Before Preregistering Data Exclusion Rules

Frequently Asked Questions About Preregistering Data Exclusions

Decide What Makes Data Unusable Before Knowing Which Exclusions Favor Your Results

How to Cite This Guide

This guide is intended to be read, shared, and used in research, teaching, and academic work. If you draw on its ideas, explanations, or other content, please acknowledge the source by citing the guide. Doing so gives appropriate credit and helps your readers locate the original resource.