Can Journal Editors Use AI to Evaluate Submitted Research Manuscripts?

Can Journal Editors Use AI to Decide Whether a Manuscript Deserves Publication?

A journal receives hundreds of manuscripts each month. Editors must determine whether submissions fit the journal's scope, satisfy reporting requirements, and warrant external peer review.

Generative AI appears capable of helping with these tasks. It can summarize manuscripts, identify possible reporting omissions, compare abstracts with journal aims, and generate preliminary assessments of methodological quality.

Some applications seem relatively administrative. Others approach the scientific judgments traditionally made by editors and peer reviewers.

Could a journal use AI to recommend which manuscripts should be desk rejected? Could it automatically identify methodological weaknesses or evaluate the significance of a study's contribution?

The central question is not merely whether AI can produce an editorial assessment. It is whether that assessment is sufficiently reliable, confidential, fair, and accountable to influence decisions affecting researchers' opportunities to publish.

AI May Support Editorial Evaluation, but It Should Not Replace Accountable Editorial Judgment

Where AI Can Assist Editorial Evaluation and Where Its Limitations Matter

Editorial Evaluation Includes Several Different Decisions

Journal editors perform multiple functions before and during peer review. Some involve checking whether a submission meets explicit requirements. Others require substantive scholarly judgment.

For example, determining whether a manuscript includes an abstract is relatively straightforward. Determining whether its research question represents an important contribution to the field is considerably more interpretive.

These tasks should not be treated as equivalent simply because an AI system can generate responses to both.

The distinction between administrative support and scientific judgment is important because different tasks require different evidence of system reliability.

AI May Be More Suitable for Explicit Administrative Requirements

Some editorial checks can be expressed through relatively clear rules.

A journal might require a structured abstract, a conflict-of-interest declaration, a data availability statement, or a particular reporting checklist.

Software can help identify whether these materials appear to be present. In some cases, conventional rule-based validation may be more dependable than generative AI.

However, identifying the presence of a statement is not the same as establishing its accuracy or adequacy.

For example, an AI system might detect a data availability statement without determining whether the stated repository actually contains the promised data.

Administrative automation should therefore be evaluated against the precise requirement it is intended to check.

Can AI Determine Whether a Manuscript Fits a Journal's Scope?

AI may help compare a manuscript's topic, research questions, and terminology with the journal's published aims and scope.

This could support preliminary screening, particularly for journals receiving submissions across several related disciplines.

However, scope assessment is not always a simple classification task.

An interdisciplinary educational technology manuscript may contribute to learning sciences, information systems, or educational policy without using the terminology most frequently associated with any one journal.

An AI system trained or configured around conventional topic patterns might incorrectly classify such work as outside the journal's scope.

Editors should therefore verify scope recommendations against the actual contribution and the journal's current editorial priorities.

Can AI Identify Reporting Problems Before Peer Review?

AI may assist with checking whether a manuscript addresses relevant reporting items.

For example, a randomized trial may be assessed against applicable CONSORT guidance, while a systematic review may be checked against PRISMA reporting requirements.

However, the appropriate reporting guideline depends on the study design, and the absence of an explicitly labeled section does not necessarily mean the required information is missing.

A manuscript may describe an important procedure under a different heading or in supplementary material.

Conversely, including a reporting item does not establish that the underlying procedure was methodologically sound.

AI-generated reporting flags should therefore be treated as prompts for verification rather than definitive findings.

Why AI-Generated Methodological Assessments Can Be Unreliable

Generative AI can identify familiar methodological concerns, including possible confounding, inadequate measurement, missing-data problems, and unsupported causal interpretation.

Nevertheless, it can also produce technically convincing criticisms that do not apply to the manuscript.

For example, an AI system might recommend statistical power calculations for an interpretive qualitative study or demand random assignment in a descriptive investigation.

These comments may reflect inappropriate assumptions about what constitutes rigorous research.

Editors should distinguish general methodological questions from verified weaknesses established through examination of the actual manuscript.

The concerns discussed in confident but irrelevant AI-generated peer-review comments apply equally when the initial assessment is produced for an editor.

Can AI Help Editors Identify Potential Research Integrity Concerns?

Editorial systems may use automated tools to flag potential problems involving text similarity, image manipulation, citation anomalies, or unusual reporting patterns.

Such tools differ substantially in their technical approaches. Not all are generative AI systems, and their outputs should not be treated as interchangeable.

A similarity report, for example, identifies textual overlap rather than independently establishing plagiarism. A statistical anomaly may warrant investigation without proving fabrication.

Generative AI may also suggest possible integrity concerns, but its ability to generate a persuasive allegation does not establish misconduct.

Editors should use appropriate investigative procedures, examine original evidence, and provide authors with a fair opportunity to respond where required.

Automated flags should not be converted into allegations of misconduct without independent substantiation.

Could AI Influence Desk-Rejection Decisions?

Desk rejection occurs when an editor declines a manuscript without external peer review, often because of scope, editorial priorities, or concerns about suitability.

AI might help identify submissions that appear outside scope or lack required materials. However, automatically rejecting manuscripts based on AI-generated assessments raises substantial fairness and accountability concerns.

For example, a model may underestimate the relevance of interdisciplinary research, unfamiliar theoretical perspectives, or studies conducted in underrepresented educational contexts.

It may also confuse unconventional research designs with methodological deficiencies.

Because desk rejection can prevent a manuscript from receiving specialist evaluation, editors should independently assess consequential recommendations rather than treat AI scores or summaries as authoritative.

Why Editorial AI Can Introduce Bias

AI systems may reflect patterns in the data and processes through which they were developed.

These patterns could favor conventional research topics, familiar academic language, dominant methodological traditions, or institutions and regions that are more extensively represented in available training material.

Such effects are possible rather than inevitable. Their extent must be evaluated empirically for the particular system and editorial task.

However, editors should not assume that an automated assessment is neutral simply because it is generated computationally.

A manuscript written in less conventional academic English may still present rigorous and important research. Similarly, a locally situated educational study may make a meaningful contribution even when its context differs from dominant international research settings.

The possibility of AI bias affecting peer-review and editorial decisions deserves attention when automated recommendations influence publication opportunities.

Confidentiality Obligations Also Apply to Journal Editors

Editors receive manuscripts as part of a confidential editorial process. Their access does not automatically authorize disclosure to any external AI provider.

Manuscripts may contain unpublished discoveries, confidential datasets, personal information, or potentially patentable material.

Using an AI system to summarize or assess a manuscript may involve transmitting this information to another processing environment.

Editors should follow the publisher's policies, contractual requirements, applicable data protection obligations, and authorized technical arrangements.

Some publishers restrict manuscript uploads to external generative AI systems while permitting selected internal or approved editorial technologies.

An institutionally managed system may offer stronger safeguards, but its availability does not automatically establish that every editorial use is authorized.

How Is Editorial AI Different From AI Used Independently by Reviewers?

A publisher may establish an authorized AI-assisted editorial workflow with defined access controls, governance procedures, and oversight.

That arrangement differs from an individual reviewer independently uploading a confidential manuscript to a general-purpose AI service.

The publisher may have contractual authority to process submissions through approved systems, subject to its policies and applicable legal obligations.

However, authorization does not establish that the AI-generated assessment is scientifically accurate or fair.

Both arrangements require attention to accuracy and accountability. The second may additionally involve unauthorized disclosure of confidential manuscript information.

Can AI Replace External Peer Review?

AI-generated assessments may resemble reviewer reports, but producing a structured critique is not equivalent to conducting independent expert peer review.

Scientific evaluation requires understanding the study's aims, methodological assumptions, evidentiary contribution, and disciplinary context.

A model may overlook consequential weaknesses while generating numerous minor or irrelevant criticisms.

It may also produce recommendations that appear sophisticated but cannot be independently substantiated.

Editors should therefore distinguish AI-generated preliminary observations from the judgments of qualified reviewers.

The limitations of automatically generated peer-review reports become particularly important when journals consider using AI to reduce reliance on external reviewers.

What Does Meaningful Human Oversight Require?

Human oversight should involve more than approving a recommendation displayed by an AI system.

The editor should understand the basis for the assessment, examine the relevant manuscript evidence, and be able to reject an unsupported recommendation.

For consequential decisions, the editor may need to consult independent reviewers or methodological specialists.

Oversight is weakened when editors routinely accept AI recommendations without examining their validity. This can create automation bias, especially when the system presents scores or classifications with an appearance of precision.

The editor's responsibility is to make a defensible decision, not merely to confirm that an automated process has produced one.

Should AI Editorial Systems Be Validated Before Deployment?

Yes. Systems used for consequential editorial tasks should be evaluated for their intended purposes before their outputs influence decisions.

Validation may involve comparing AI-generated assessments with independently established reference decisions, examining false positives and false negatives, and testing performance across manuscript types.

For example, an AI scope-screening system could be evaluated on interdisciplinary submissions, manuscripts from different regions, and studies using diverse methodological approaches.

Editors should also consider whether the system's performance changes over time or after model updates.

Agreement with historical editorial decisions may be informative, but it is not automatically a valid reference standard. Historical decisions may themselves reflect inconsistent practices or biases.

Validation should therefore examine substantive correctness and fairness rather than simply reproduce past editorial patterns.

Why Monitoring Matters After Deployment

An AI system that performs adequately during initial testing may behave differently after updates, changes in submission patterns, or modifications to its prompts and processing environment.

Journals should consider monitoring error patterns, disputed assessments, and performance across relevant manuscript categories.

Consequential changes to the system should be documented, and renewed evaluation may be necessary when those changes affect its intended use.

Monitoring should also examine whether editors are becoming excessively dependent on AI recommendations.

The objective is not simply to measure how frequently the system agrees with editors, but whether it continues to support fair and scientifically defensible editorial decisions.

Should Authors Know When AI Was Used in Editorial Assessment?

This raises a separate transparency question.

Authors may reasonably want to understand whether AI contributed to manuscript screening, methodological assessment, or an editorial decision.

However, publisher policies differ in how they communicate the use of automated systems. Some administrative tools may be treated differently from AI systems that materially influence scientific judgment.

The appropriate disclosure arrangement should consider the nature of the assistance, its consequences, and the journal's transparency obligations.

The question of whether authors should be informed about AI-assisted editorial assessment becomes particularly important when automated recommendations influence rejection or revision decisions.

Who Is Accountable for an AI-Assisted Editorial Decision?

Editorial responsibility remains with the journal's authorized decision-makers.

An AI system cannot independently justify a rejection, respond to an author's appeal, or assume responsibility for unfair or inaccurate assessments.

Editors should be able to explain the substantive basis for decisions and distinguish verified concerns from automated suggestions.

When AI contributes to the process, appropriate records may be necessary to establish what the system recommended and how the editor evaluated that recommendation.

The broader principle is that technological assistance should support, rather than displace, accountable scholarly decision-making.

When AI Incorrectly Recommends Desk Rejection

The example illustrates why an AI-generated scope recommendation should not automatically determine whether a manuscript receives peer review.

A system may classify a manuscript according to familiar terminology while overlooking the disciplinary contribution that makes the work relevant.

Human oversight is meaningful when editors can independently evaluate and overturn such recommendations.

Misconceptions About AI in Editorial Manuscript Evaluation

How Journals Can Establish Appropriate Boundaries for AI Editorial Assistance

Editors and publishers should distinguish tasks that can be checked against explicit requirements from those requiring interpretive scholarly judgment.

They should also establish the system's permitted purpose, confidentiality safeguards, validation procedures, and mechanisms for reviewing consequential recommendations.

Evaluate AI Systems Against the Actual Editorial Task

A system designed to identify missing documents should not be assumed capable of evaluating scientific originality.

Similarly, strong performance in one discipline does not establish equivalent performance across other research traditions or manuscript types.

Journals should assess the system under conditions representative of their submissions, including interdisciplinary work and less conventional methodological approaches.

They should also monitor whether system updates change the reliability of its recommendations.

The appropriate boundary is not simply between using AI and avoiding it. It is between assistance that improves an authorized editorial process and automation that obscures responsibility for scientific judgments.

Checklist for AI-Assisted Editorial Manuscript Evaluation

Frequently Asked Questions About AI in Journal Editorial Assessment

AI May Support Editorial Work, but It Should Not Become the Editor

How to Cite This Guide

This guide is intended to be read, shared, and used in research, teaching, and academic work. If you draw on its ideas, explanations, or other content, please acknowledge the source by citing the guide. Doing so gives appropriate credit and helps your readers locate the original resource.