P-Hacking, HARKing, and Selective Reporting: Research Practices You Need to Avoid

When Does Flexible Research Become Misleading Research?

Real research rarely unfolds exactly as planned. You may discover an unexpected pattern, realize that another statistical model is more appropriate, examine a subgroup you had not considered, or develop a new hypothesis after seeing the data. None of those actions is automatically improper.

The problem begins when the published account conceals how those decisions were made. If you try analyses until a favorable result appears, present a hypothesis developed after seeing the results as though you predicted it beforehand, or report only the outcomes that support the preferred story, readers receive a distorted picture of the evidence.

Three terms appear frequently in discussions of these problems: p-hacking, HARKing, and selective reporting. They overlap, but they are not interchangeable. Understanding the difference matters because the remedy is not to eliminate exploration from research. It is to prevent exploration and analytical flexibility from masquerading as stronger confirmatory evidence than they actually provide.

What Are P-Hacking, HARKing, and Selective Reporting?

How These Practices Distort the Research Record

P-Hacking Exploits Analytical Flexibility in Search of a Significant Result

Researchers routinely make analytical decisions. They may need to decide which observations to exclude, whether to include covariates, how variables should be operationalized, which statistical model is appropriate, whether transformations are needed, and which subgroups warrant investigation. Such choices are often called researcher degrees of freedom.

Having these choices is not itself a problem. The problem arises when choices become contingent on obtaining a desirable result and that contingency is hidden from the reader.

For example, suppose an analysis produces p =.08. A researcher then removes several observations and obtains p =.06, adds a covariate and obtains p =.04, and reports only the final model. The reported analysis may look like a single planned test even though it was effectively selected from several alternatives because it crossed the significance threshold.

This is why what p-hacking looks like in practice can be subtler than simply manipulating data. Simmons, Nelson, and Simonsohn demonstrated how undisclosed flexibility involving dependent variables, sample size, covariates, and subsets of conditions can substantially increase the probability of obtaining false-positive findings.

Importantly, trying several analyses is not automatically p-hacking. Sensitivity analyses, robustness checks, model comparisons, exploratory analyses, and analyses required to investigate violated assumptions may all be legitimate. What matters is why the alternatives were examined, how inferential uncertainty is handled, and whether the analytical process is reported honestly.

HARKing Rewrites When a Hypothesis Was Developed

HARKing stands for Hypothesizing After the Results are Known, a term introduced by Norbert Kerr. The critical feature is not simply developing a hypothesis after examining data. Scientific discoveries often generate new hypotheses. HARKing occurs when a post hoc hypothesis is presented in a research report as though it had been an a priori hypothesis.

Imagine that you predicted that intervention A would improve outcome X. The result for X is null, but you unexpectedly find an association with outcome Y. You develop a plausible theoretical explanation for Y and write the manuscript as though the study had been designed from the beginning to test that prediction. The final paper erases the distinction between prediction and discovery.

That distinction matters because evidence used to generate a hypothesis is not equivalent to an independent test of that same hypothesis. A pattern discovered in one dataset may reflect a genuine phenomenon, but it may also reflect sampling variability or the many possible patterns that could have attracted attention. A new dataset can provide a stronger confirmatory test.

The appropriate response is therefore not to suppress unexpected findings. Researchers can report exploratory findings without pretending they were predicted. The problem is the retrospective reconstruction of the research story.

Selective Reporting Changes Which Evidence Readers Get to See

Selective reporting occurs when the reported evidence is chosen in a way that produces an incomplete or distorted representation of what was investigated or found. The selection can occur at several levels.

A study might measure five outcomes but emphasize only the one that produces a favorable result. A researcher might conduct several statistical models but report only the successful specification. A manuscript might describe a subgroup effect while leaving unreported the unsuccessful subgroup analyses that led to it. In intervention research, prespecified outcomes may be omitted, altered, or replaced in the published report.

Not every omission is selective reporting in the problematic sense. Papers have finite space, some analyses may genuinely be redundant, and exploratory work can generate more output than belongs in a journal article. The central issue is whether what has been omitted would materially change a reasonable reader's interpretation of the evidence.

This is particularly important when deciding whether to leave a nonsignificant analysis out of a paper. Statistical significance should not itself determine whether a relevant analysis deserves to be reported.

The Three Practices Can Occur Together

These practices can form a sequence. A researcher might try numerous analyses until one produces a statistically significant result, construct a hypothesis that explains that result, and then publish only the successful analysis and newly constructed hypothesis. P-hacking affected the analytical search, HARKing affected how the hypothesis was portrayed, and selective reporting affected what the reader ultimately saw.

The Central Problem Is Hidden Data-Dependent Decision-Making

Many analytical choices must legitimately be made after researchers learn something about their data. A distribution may violate model assumptions. Measurement problems may become apparent. Missing data may require an alternative strategy. An unexpected result may suggest a scientifically valuable follow-up analysis.

Accordingly, a simple rule such as “never change your analysis after seeing the data” is not realistic. A more useful question is whether a decision was data-dependent, whether it affects the strength or interpretation of the inference, and whether readers are told enough to understand that dependency.

This is where analytical flexibility can become a questionable research practice. Flexibility becomes especially concerning when researchers repeatedly exploit it in the direction of preferred findings while presenting the eventual choice as inevitable or prespecified.

Exploratory and Confirmatory Research Serve Different Purposes

Both are valuable. Problems arise when the distinction disappears in reporting.

An exploratory finding may be surprising, theoretically interesting, and worthy of publication. Labeling it exploratory does not make it scientifically inferior. It tells readers how the finding arose and therefore how strongly it should be interpreted as a test of a prior prediction.

Conversely, merely calling an analysis “confirmatory” does not make it rigorous. Its credibility also depends on study design, measurement, statistical assumptions, data quality, adherence to the analysis plan, and appropriate interpretation.

Preregistration Can Help, but It Does Not Replace Judgment

One way to make the distinction between planned and data-dependent decisions more visible is to document hypotheses, outcomes, exclusion criteria, sampling decisions, and analysis plans before observing the relevant results. Depending on the field and study design, this may take the form of a protocol, preregistration, statistical analysis plan, or registered report.

Prespecification creates a record against which later deviations can be identified. It does not mean researchers are forbidden from changing course. Unexpected complications are normal. A transparent report can explain what changed, why it changed, and which analyses were prespecified versus added later.

Nor does preregistration guarantee good research. A poorly designed prespecified analysis remains poorly designed. Its main value here is transparency about the temporal order of decisions, not a magical conversion of methodological choices into correct ones.

How an Ordinary Analysis Can Turn Into a Misleading Research Story

Several different integrity problems now coexist. Searching through analytical alternatives for a favorable significance result may constitute p-hacking, depending on how the analyses were conducted and interpreted. Presenting the newly developed confidence hypothesis as a prior prediction is HARKing. Omitting relevant outcomes and unsuccessful analyses can become selective reporting if those omissions materially distort the overall evidence.

The same unexpected confidence result could have been reported responsibly. The researcher could state that the primary analysis did not support the original hypothesis, identify the confidence analysis as exploratory, disclose relevant analytical multiplicity, and present the finding as a hypothesis worth testing independently.

The data have not changed. What changes is the honesty of the inference.

Common Misunderstandings About Questionable Research Practices

How to Explore Your Data Without Misrepresenting the Evidence

You do not need to eliminate analytical flexibility. You need to make consequential flexibility visible and interpret the resulting evidence accordingly.

The same principle applies when results complicate the story you hoped to tell. Researchers are expected to interpret evidence, not merely dump output into a manuscript, so some selection and organization are unavoidable. The relevant boundary is whether telling a clear research story becomes misrepresentation.

A useful test is to imagine that another competent researcher receives your complete analysis history. Would that person interpret the strength, consistency, and confirmatory status of your evidence substantially differently from a reader of the paper? If so, the manuscript may be hiding information that matters.

Transparency does not make a messy study magically rigorous. It does, however, allow readers to evaluate the study that was actually conducted rather than a cleaner study reconstructed after the fact.

Before You Finalize Your Analysis and Results

Questions Researchers Ask About P-Hacking, HARKing, and Selective Reporting

Explore Freely, but Report the Research That Actually Happened

How to Cite This Guide

This guide is intended to be read, shared, and used in research, teaching, and academic work. If you draw on its ideas, explanations, or other content, please acknowledge the source by citing the guide. Doing so gives appropriate credit and helps your readers locate the original resource.