03 · What You Need to Know
How to Distinguish Research Errors From Unexpected Scientific Findings
What Makes a Research Result Unusual?
A research result may be considered unusual because it differs from a theoretical prediction, contradicts previous studies, departs from an expected statistical pattern, or appears inconsistent with other measurements.
These are different forms of unexpectedness.
A statistically extreme observation may be unusual relative to a distribution. A negative association may be unexpected because the researcher predicted a positive relationship. A treatment effect may be surprising because previous studies reported little evidence of benefit.
None of these circumstances independently establishes an error.
Researchers should first identify the basis for describing the result as unusual. Is the concern statistical, methodological, theoretical, or related to data quality?
This distinction determines which verification procedures are relevant.
Unexpected Results and Research Errors Are Not Equivalent
An error involves a problem in data collection, measurement, recording, processing, analysis, or interpretation that causes the research record or conclusion to depart from what the applicable procedure and evidence support.
An unexpected result, by contrast, may be produced by a correctly implemented study and analysis.
Research error
A demonstrable problem with the evidence, procedure, calculation, or interpretation that requires correction or qualification.
Unexpected finding
A result that differs from a prediction, expectation, or previous observation but may nevertheless be supported by valid evidence.
For example, a negative regression coefficient resulting from an incorrectly reverse-coded variable is an analytical error. A negative coefficient obtained from correctly processed data is not automatically erroneous, even if the researcher expected a positive association.
Researchers should avoid defining error in relation to whether the finding confirms their expectations.
What Types of Errors Can Produce Unusual Results?
Unexpected findings may arise from problems at different stages of the research process.
| Potential source |
Illustrative problem |
Evidence to examine |
| Data entry |
A measurement of 12 is recorded as 120 |
Original forms, instrument records, and data-entry logs |
| Variable coding |
A reverse-scored questionnaire item is not recoded |
Codebook, scoring instructions, and transformation code |
| Data merging |
Records are matched using incorrect participant identifiers |
Merge keys, source datasets, and reconciliation reports |
| Measurement |
An instrument is improperly calibrated |
Calibration records and measurement procedures |
| Statistical analysis |
An inappropriate model or incorrect formula is applied |
Analysis scripts, assumptions, and independent calculations |
| Interpretation |
A correlation is described as a causal effect |
Study design, estimand, and inferential assumptions |
AI may help researchers develop a systematic investigation plan for these possibilities.
However, identifying a plausible source of error is not the same as demonstrating that it occurred.
Why AI May Incorrectly Treat Surprising Findings as Mistakes
Generative AI systems produce responses influenced by their training, instructions, and the information provided in the prompt.
When asked to interpret a surprising result, a model may generate explanations consistent with common research patterns or widely discussed theories.
This can be helpful for identifying possibilities, but it may also encourage an unwarranted preference for conventional findings.
For example, if a researcher asks why an intervention produced an unexpected negative effect, AI might suggest that the dataset contains errors before adequately considering that the intervention may have been ineffective or harmful in the studied context.
The system may also present an explanation confidently without having access to the original data or sufficient information about the research design.
Researchers should therefore distinguish an AI-generated hypothesis about a possible error from evidence that the error actually exists.
Statistical Significance Does Not Determine Whether a Result Is Correct
A statistically significant result can be erroneous, and a nonsignificant result can arise from correctly collected and analyzed data.
A p-value is calculated under a specified statistical model and its assumptions. It does not directly measure the probability that a scientific hypothesis is true or that the data are error-free.
The American Statistical Association's statement on p-values emphasizes that scientific conclusions should not depend solely on whether a p-value crosses a particular threshold.
Researchers should examine effect estimates, uncertainty intervals, study design, statistical assumptions, and the substantive meaning of the findings.
AI should not label a result erroneous because its p-value is unexpected, nor should it treat statistical significance as proof of validity.
The separate question of whether AI should determine statistical significance requires attention to the prespecified analysis and the interpretation of inferential evidence.
An Unusual Observation May Be Genuine and Influential
A surprising result may depend on a small number of unusual observations. This possibility warrants investigation, but it does not automatically justify removing those observations.
Consider a study in which a few participants report exceptionally high levels of engagement with an online learning platform. Their outcomes may differ substantially from those of other participants.
These observations might reflect recording errors, atypical usage patterns, or genuine differences in participant experiences.
Researchers should examine the relevant source evidence and statistical diagnostics before deciding how to treat them.
AI-assisted statistical outlier detection and removal should therefore remain distinct from the determination that an entire research result is erroneous.
Deleting valid observations merely because they weaken an expected relationship can introduce bias.
Unexpected Findings May Reveal Problems With Model Specification
An unusual result may arise because the analytical model does not adequately represent the data-generating process.
Potential concerns include omitted variables, inappropriate functional forms, violations of statistical assumptions, measurement error, or failure to account for clustering and repeated observations.
For example, a researcher might fit a simple linear model to a relationship that is actually nonlinear. The resulting coefficient could obscure meaningful variation across different levels of the predictor.
AI may help identify diagnostic procedures, but the researcher must evaluate whether the proposed alternative model is theoretically and statistically appropriate.
A different model should not be selected merely because it produces a result more consistent with expectations.
The choice of which statistical model becomes the final model requires independent justification, particularly when alternatives are explored after observing the initial results.
Confounding Can Produce Surprising Associations Without Data Errors
In observational research, an unexpected association may reflect confounding rather than incorrect measurement or calculation.
Suppose students who are struggling academically use a tutoring platform more frequently because they need additional assistance.
A cross-sectional analysis might show that higher platform usage is associated with lower examination performance.
This negative association does not necessarily mean the platform causes poor performance. Students' prior difficulties may influence both their platform use and their subsequent outcomes.
The observed relationship could therefore reflect selection into platform use, reverse causation, confounding, or a combination of mechanisms.
Appropriate covariate selection for causal analysis may help address some concerns, but causal conclusions remain dependent on the study design and assumptions.
The finding may be statistically correct while its initial causal interpretation is inappropriate.
Measurement Problems Can Produce Results That Look Scientifically Meaningful
Researchers should also consider whether the variables adequately represent the constructs being studied.
An educational technology study might use login frequency as a measure of learning engagement. Yet frequent logins could reflect productive learning, difficulty navigating the platform, repeated technical interruptions, or other activities.
A negative relationship between login frequency and examination performance may therefore be surprising only because the researcher assumed that logins were an unambiguous measure of engagement.
The issue may concern construct validity rather than an erroneous statistical calculation.
AI can suggest alternative interpretations, but these should be assessed against the instrument design, relevant literature, and available evidence.
Researchers should not automatically replace an inconvenient operational definition after observing the results without documenting the implications.
Reproducibility Checks Can Help Identify Computational Errors
One practical response to an unexpected finding is to determine whether the result can be reproduced from the original data and documented analytical procedure.
Researchers may rerun scripts, verify transformations, inspect software settings, and independently calculate important statistics.
For example, an unexpected regression coefficient might be traced to an incorrect reference category, a reversed coding scheme, or a data merge that duplicated records.
AI can assist by reviewing code or proposing test cases, but its suggestions should be checked through execution and comparison with authoritative software documentation.
Reproducing the same numerical result does not establish that the model or research design is valid. It establishes that the computational procedure produces the result under the specified conditions.
Researchers must distinguish computational reproducibility from broader scientific validity.
Independent Verification Is More Reliable Than Repeated AI Reassurance
Asking an AI system repeatedly whether a result is correct may generate different explanations without resolving the underlying uncertainty.
The same model may reproduce its original assumptions, and another model may share similar limitations.
Meaningful verification should involve independent evidence, such as original measurements, data dictionaries, analytical scripts, validated statistical procedures, or relevant disciplinary expertise.
When the result affects participant safety or other consequential decisions, researchers should establish an appropriate level of independent human verification.
The reviewer should have the competence and information necessary to challenge both the AI recommendation and the researcher's original interpretation.
Replication and Triangulation Can Strengthen Confidence in Unexpected Findings
When a result remains unusual after data and analytical checks, additional evidence may help clarify its meaning.
Replication examines whether a finding can be observed again under relevant conditions. Triangulation may involve investigating a question through different data sources, measures, designs, or analytical approaches.
These strategies can strengthen an interpretation, but they do not guarantee that a result is correct. Replication may be affected by shared methodological limitations, while different approaches may depend on related assumptions.
Researchers should also distinguish direct replication from conceptual replication and from reanalysis of the same dataset.
A surprising result that has not yet been independently replicated may still be worth reporting, provided that uncertainty and limitations are described accurately.
Do Not Revise the Analysis Merely to Restore the Expected Result
Unexpected findings may tempt researchers to try alternative exclusions, transformations, covariate sets, or statistical models until the result agrees with the original hypothesis.
Such practices can introduce selective reporting and undermine the interpretation of conventional statistical inference.
Exploratory analysis is legitimate when it is transparently distinguished from prespecified confirmatory analysis.
Researchers should document why additional analyses were performed and report consequential departures from the original analytical plan.
An AI recommendation does not make an outcome-driven analytical change methodologically defensible.
Watch Out
Do not treat agreement with the original hypothesis as a data-quality criterion. A result should not be corrected, excluded, or replaced simply because it contradicts theoretical expectations or previous findings.
When Should an Unusual Result Be Corrected, Qualified, or Retained?
The appropriate response depends on what the investigation establishes.
If an error is confirmed and a reliable correction is available, researchers should correct the affected data or analysis and preserve an audit trail.
If a methodological limitation is identified but cannot be eliminated, the finding may require qualification, alternative analysis, or a narrower interpretation.
If no error is established, the result should not be discarded solely because it is surprising.
Researchers may report the unexpected finding with appropriate uncertainty and explain plausible interpretations without presenting speculation as established fact.
The final judgment about whether the evidence supports the research hypothesis should follow from the design, analysis, and strength of the evidence rather than AI-generated expectations.
