03 · What You Need to Know
What AI Can and Cannot Reliably Check in Statistical Peer Review
Statistical Verification Involves Different Kinds of Questions
Checking a statistical claim is not one activity. A reviewer may need to verify arithmetic, assess whether an analysis is appropriate, examine the interpretation of an estimate, or determine whether the evidence supports a conclusion.
These tasks require different forms of evidence.
For example, checking whether a reported percentage was calculated correctly may require straightforward arithmetic. Evaluating whether a regression coefficient supports a causal interpretation requires understanding the research design and its assumptions.
Generative AI may be useful for formulating questions about both, but its generated explanation does not establish that the answer is correct.
| Statistical Task |
Possible AI Contribution |
Required Verification |
| Arithmetic consistency |
Suggest calculations worth checking |
Independent calculation using reliable software |
| p-value interpretation |
Identify potentially misleading statements |
Examine the test, assumptions, and reported evidence |
| Confidence intervals |
Suggest whether the interpretation matches the interval |
Verify interval construction and parameter meaning |
| Regression analysis |
Raise questions about assumptions or interpretation |
Examine model specification and available diagnostics |
| Statistical significance |
Identify possible overstatements |
Evaluate effect size, uncertainty, and study design |
| Causal inference |
Suggest alternative explanations |
Assess identification assumptions and research design |
The distinction matters because a correct general explanation of a statistical concept does not establish that the concept has been applied correctly to a particular manuscript.
Can AI Detect Incorrect p-Value Interpretations?
Generative AI may recognize common misconceptions about statistical significance. For example, it may identify statements that incorrectly treat a p-value as the probability that the null hypothesis is true.
Under a specified statistical model and null hypothesis, a p-value concerns how incompatible the observed data, or more extreme results as defined by the test, are with the null model. It is not a direct probability that the hypothesis is correct or incorrect.
The American Statistical Association's statement on p-values emphasizes that statistical significance does not measure the size or importance of an effect and should not serve as the sole basis for scientific conclusions.
AI may help reviewers recognize these issues, but the reviewer must determine whether the manuscript actually makes the alleged interpretive error.
Why Statistical Significance Does Not Establish Practical Importance
A statistically significant finding can have a small practical effect, particularly in studies with large samples or precise estimates.
Conversely, a study may estimate a potentially important effect but lack sufficient precision to establish whether the result is incompatible with a specified null hypothesis at a conventional threshold.
For example, an educational technology intervention might produce a statistically detectable improvement in assessment scores that is too small to justify the financial and instructional costs of implementation.
AI may suggest that a reviewer examine effect sizes and confidence intervals rather than focus exclusively on p-values.
However, judging practical importance requires substantive knowledge of the educational setting, measurement scale, intervention costs, and research objectives.
Can AI Identify Misinterpreted Confidence Intervals?
Generative AI may recognize common problems in confidence interval interpretation, but its explanations require verification.
For a conventional frequentist 95% confidence interval, the nominal 95% coverage refers to the long-run performance of the interval-generating procedure under its assumptions.
It does not ordinarily mean that a fixed population parameter has a 95% probability of lying inside a particular calculated interval.
Reviewers should also examine whether the interval corresponds to the parameter being discussed, whether its construction is appropriate, and whether the authors interpret its width and direction correctly.
A confidence interval can communicate information about estimation uncertainty that is obscured by a binary significant-versus-nonsignificant interpretation.
Can AI Check Whether a Statistical Test Was Appropriate?
AI may suggest questions about the relationship between variable types, research design, and statistical procedures.
For example, a reviewer may need to determine whether observations are independent, whether repeated measurements were appropriately modeled, or whether clustering within classrooms was accounted for.
In educational research, students may be nested within classes, instructors, or institutions. Ignoring this structure can lead to inappropriate uncertainty estimates under some analytical conditions.
However, AI may also make oversimplified recommendations, such as insisting that every dataset with non-normal observations requires a nonparametric test.
Statistical assumptions depend on the particular model and inferential procedure. They cannot be evaluated reliably through generic rules alone.
Reviewers should therefore verify any suggested criticism against the actual analytical specification and appropriate methodological literature.
Why AI Can Miscalculate Statistical Results
Language models generate responses using learned patterns and inference procedures. Their ordinary text-generation mechanisms are not equivalent to a verified statistical computation engine.
They may make errors in arithmetic, apply the wrong formula, confuse standard deviations with standard errors, or misinterpret the information needed for a statistical test.
For numerical verification, conventional statistical software is generally preferable because the calculation can be specified explicitly, tested, and reproduced.
AI-generated code can sometimes assist with implementing such calculations, but that code must be independently reviewed and tested.
A convincing explanation accompanying an incorrect calculation does not make the result more reliable.
Can Reviewers Recalculate a Reported Statistic?
Some reported statistics can be checked from sufficient summary information. Others require access to individual-level data, model specifications, or computational scripts.
For example, a reviewer may be able to recalculate a simple percentage from the reported numerator and denominator.
However, reconstructing a regression analysis may require the original dataset, variable coding, missing-data handling, estimation method, and software settings.
When sufficient information is unavailable, the reviewer should request clarification or additional reporting rather than assert that the calculation is incorrect.
Even when AI proposes a plausible reconstructed value, it should not be presented as verified unless the necessary assumptions and calculations have been established.
What About Checking Statistical Consistency Across a Manuscript?
A manuscript may report one sample size in the methods section, another in a results table, and a third in a figure caption.
Similarly, the abstract may describe a result differently from the main analysis.
These discrepancies may warrant investigation. However, different sample sizes can sometimes be legitimate because analyses use different subsets, exclusions, or missing-data procedures.
AI may help formulate a consistency-checking framework when permitted, but reviewers must verify whether apparent discrepancies represent errors or justified analytical differences.
The distinction between incomplete reporting and incorrect analysis remains important.
Can AI Identify Problems With Multiple Testing?
Generative AI may suggest examining whether a study conducted numerous statistical tests without adequately addressing the risk of misleading findings.
However, not every collection of tests requires the same correction procedure. The appropriate approach depends on the inferential goals, whether hypotheses were prespecified, the relationships among tests, and the intended interpretation.
For example, a confirmatory analysis involving a defined family of hypotheses may require different considerations from an explicitly exploratory investigation.
Reviewers should avoid automatically demanding a particular multiple-comparison correction without establishing why it is appropriate.
Can AI Evaluate Statistical Claims About Causation?
A statistically significant association does not, by itself, establish causation.
Suppose students who use a learning management system more frequently obtain higher grades. The association may reflect the effects of platform engagement, but it could also involve prior achievement, motivation, access to technology, or other factors.
AI may suggest considering confounding and alternative explanations. Yet identifying plausible confounders is not equivalent to establishing their causal roles or determining whether the manuscript's analytical strategy addresses them adequately.
The reviewer must evaluate the study design, identification assumptions, and the scope of the authors' claims.
This connects statistical review to the broader task of identifying methodological weaknesses during peer review.
Why Confidentiality Limits AI-Assisted Statistical Checking
Statistical checking may require tables, figures, unpublished results, or supplementary datasets. These materials are generally part of the confidential manuscript record.
Reviewers should not upload them to an external AI service without the journal's required authorization and appropriate safeguards.
The ICMJE advises reviewers to follow the journal's AI policy or request permission before using AI to facilitate peer review. It also emphasizes maintaining manuscript confidentiality.
Some publishers impose stricter restrictions on uploading manuscript content to generative AI systems.
Reviewers should therefore establish whether confidential manuscript information may be processed by an AI tool before attempting manuscript-specific statistical evaluation.
A general statistical question that contains no confidential information may be treated differently, depending on the journal's policy.
What Makes an AI-Assisted Statistical Criticism Defensible?
A defensible criticism identifies the specific statistical claim, explains the concern, and connects it to the appropriate evidence or methodological standard.
For example, a reviewer might observe that the authors interpret a nonsignificant comparison as proof that two interventions are equivalent, even though the study did not use an equivalence-testing framework.
The criticism would be grounded in the distinction between failing to reject a null hypothesis and demonstrating equivalence within a justified margin.
By contrast, simply stating that "the statistical analysis is inappropriate" without explaining why provides little useful guidance.
Reviewers should be able to understand and defend every criticism independently. If specialized expertise is needed, the handling editor may need to seek an additional statistical review.
