How Do You Know Whether Your Inclusion Criteria Were Shaped by the Results You Found?

Did Your Criteria Change Because You Learned More or Because You Saw the Results?

Literature reviews rarely unfold exactly as imagined. You discover terminology you did not anticipate, encounter unexpected study designs, realize that a population category is ambiguous, or learn that your original eligibility rule does not work well in practice.

Changing an inclusion criterion is therefore not automatically a methodological failure. Sometimes refinement is necessary.

The harder question is why the criterion changed. If the modification occurred after you knew which studies it would include or exclude, and especially if it conveniently improves the evidential story, you need to examine whether the decision was driven by a genuine methodological problem or by the results you had already seen.

Look at the Timing, Rationale, and Consequences of the Change

How Results Can Quietly Reshape Eligibility Criteria

Inclusion criteria determine what evidence gets a voice

Eligibility criteria define the boundaries of a systematic review. They specify which studies are sufficiently relevant to the question to enter the evidence base and which are not.

For intervention reviews, these decisions commonly concern populations, interventions, comparators, study designs, and other characteristics derived from the review question. Current Cochrane guidance emphasizes that criteria should be defined clearly and justified, while PRISMA 2020 requires systematic-review reports to specify their inclusion and exclusion criteria and explain how studies were grouped for synthesis.

The importance of these criteria is easy to underestimate. Once a study is excluded, its result generally cannot influence the synthesis. A seemingly technical eligibility decision can therefore change the apparent direction, consistency, and certainty of the evidence.

Pre-specification reduces your freedom to move the boundary after seeing the evidence

One reason systematic reviews use protocols is to establish important methodological decisions before the study findings are fully known. Cochrane describes the protocol as a public record of the intended question and methods and notes that advance planning can reduce opportunities for bias.

This does not make the original criteria sacred. It creates a reference point.

Without that reference point, it can become difficult to distinguish “this was always outside our intended scope” from “we decided this was outside the scope after discovering what it found.”

The timing of the decision is an important clue

Ask when you decided on the criterion relative to when you learned the studies' results.

A decision made before screening or before the relevant findings were known is less vulnerable to result-driven influence. A decision introduced after reading the results deserves more scrutiny because you can no longer unknow what effect the change will have.

Timing alone does not prove bias. You may encounter a genuinely unforeseen methodological problem only after reading several full texts. But once the consequences of a decision are visible, stronger justification and transparent documentation become particularly important.

Ask whether the new criterion conveniently separates favorable from unfavorable studies

Suppose your review initially includes studies lasting at least four weeks. After screening, you notice that studies shorter than eight weeks tend to report little benefit, whereas longer studies are more favorable. You then decide that interventions need at least eight weeks to be “meaningful.”

Perhaps there is a substantive reason why eight weeks is genuinely necessary. But because you already know what the duration threshold does to the evidence, you should not treat the revised criterion as though it had always been obvious.

A particularly strong warning sign is directional asymmetry. If each new eligibility refinement happens to remove inconvenient studies while preserving favorable ones, examine the pattern rather than evaluating each change in isolation.

Use the reversal test

A simple counterfactual can expose result-driven reasoning:

If the studies affected by this criterion had produced the opposite results, would I still want to change the rule?

Imagine that you want to exclude studies using self-reported outcomes because several of them contradict your expected conclusion. Would you still regard self-report as disqualifying if those studies strongly supported the conclusion instead?

If the methodological rationale becomes less compelling after that reversal, the criterion may be doing argumentative work rather than methodological work.

Unexpected issues really can justify post hoc changes

Rigidly refusing to change a flawed criterion is not good methodology either. Cochrane explicitly recognizes that unexpected issues can arise and permits sensible post hoc decisions, while requiring changes to eligibility criteria to be justified and documented. Its MECIR standard states that such changes should remain faithful to the review objectives and must not be based on study or synthesis findings.

Cochrane also acknowledges that review questions may sometimes need refinement as researchers gain a fuller understanding of the evidence. The guidance warns, however, that data-driven changes can introduce bias and says revisions to the review question should be documented.

The lesson is not “never change anything.” It is “do not conceal the difference between what was planned and what was learned along the way.”

A legitimate change should have a rationale that survives the results

Suppose your protocol refers to “university students,” but during screening you encounter studies combining secondary-school pupils, undergraduates, and postgraduate students without reporting separate results. You now need a rule for mixed populations that the original protocol did not specify.

That is a genuine methodological ambiguity. You can develop a rule that remains aligned with the review question, apply it consistently to all affected studies, document the amendment, and explain why it was necessary.

The justification does not depend on whether the mixed-population studies report favorable or unfavorable effects. That independence is an important sign that the change is methodologically motivated.

Changing the interpretation of a criterion can matter as much as changing its wording

Researchers sometimes leave the formal criteria unchanged while quietly interpreting them differently.

For example, “higher education students” may be interpreted narrowly for one study and broadly for another. “Validated measure” may suddenly acquire a stricter meaning when a contradictory study appears. “Relevant intervention” may become flexible enough to retain a supportive paper that otherwise sits at the edge of the review's scope.

The methodological problem is the same. Consistency concerns how the rule is applied, not merely what the protocol says.

Changing criteria should leave a visible methodological trail

If a systematic review departs from its protocol, the change should not disappear during manuscript writing. Cochrane requires protocol changes to be documented and reported, and recommends sensitivity analyses where possible to examine the impact of deviations.

Transparency is not an admission that the review failed. It allows readers to distinguish planned methods from adaptive decisions and to judge whether those adaptations plausibly affected the conclusions.

This principle becomes especially important when you later document changes made during the literature search. A polished final method should not create the false impression that every useful decision was known from the beginning.

When a Sensible-Looking Criterion Appears at a Convenient Time

The duration criterion may ultimately be defensible. What makes the situation methodologically sensitive is that the reviewers already know which findings the new boundary will remove. Transparency and analysis of the consequences are therefore more informative than pretending the decision was obvious from the start.

Common Misunderstandings About Changing Inclusion Criteria

Interrogate the Reason for Every Important Post Hoc Change

If your eligibility criteria evolved, do not immediately assume either that the change was biased or that it was harmless. Reconstruct the decision.

You should also examine whether changes to your search behaved similarly. Altering terminology after learning what the literature looks like can be legitimate, but it can also produce confirmation bias through changing search terms.

Finally, compare how your criteria affect evidence in different directions. If methodological rules repeatedly preserve favorable studies and remove unfavorable ones, you may have moved beyond ordinary adaptation toward cherry-picking the literature.

Check Whether Your Eligibility Rules Followed the Question or the Results

Frequently Asked Questions About Post Hoc Inclusion Criteria

Your Criteria Can Evolve, but Their History Should Not Disappear

How to Cite This Guide

This guide is intended to be read, shared, and used in research, teaching, and academic work. If you draw on its ideas, explanations, or other content, please acknowledge the source by citing the guide. Doing so gives appropriate credit and helps your readers locate the original resource.