03 · What You Need to Know
Why Generative AI Can Create an Appearance-of-Rigor Problem
Research Rigor and the Appearance of Rigor Are Different Things
Scientific writing contains many visible signals that readers associate with careful scholarship: technical terminology, explicit limitations, coherent theoretical arguments, methodological justification, statistical detail, citations, structured reasoning, and cautious academic language.
Those signals often accompany good research because competent researchers use them to describe genuine methodological work.
But the signals are not the work itself.
Actual rigor
The research question, design, evidence, methods, validation, analysis, interpretation, transparency, and conclusions are defensible.
Signals of rigor
The manuscript uses the language, structure, detail, and presentation readers commonly associate with rigorous scholarship.
Generative AI is exceptionally capable of producing the second category. Whether the first category improves depends on what actually happened in the research.
Generative AI Can Repair Prose Without Repairing the Study
Suppose a survey used a convenience sample that poorly represents the target population. AI can improve the description of the sampling procedure, make the limitations section more sophisticated, and produce a polished discussion of generalizability.
The sample has not changed.
Suppose a researcher used an instrument without adequate evidence that it measures the intended construct. AI can write an elegant paragraph about construct validity.
The measurement problem remains.
Suppose an observational study makes an unsupported causal claim. AI can transform the explanation into fluent academic prose while preserving the same inferential error.
This is why research rigor should be evaluated from the research process, not inferred from the professionalism of the manuscript.
AI Can Supply Methodological Vocabulary That the Research Never Operationalized
Ask a generative AI system to “make this methodology sound more rigorous,” and it may introduce terms associated with established quality practices.
A qualitative manuscript may suddenly discuss reflexivity, triangulation, saturation, member checking, audit trails, or credibility. A quantitative manuscript may acquire language about robustness, sensitivity analysis, construct validity, confounding, statistical power, or diagnostic testing.
These concepts can be entirely appropriate when they describe what researchers actually did.
They become misleading when the terminology is added retrospectively without corresponding procedures.
Watch Out
Methodological terminology is descriptive only when the underlying procedure actually occurred. Adding the word “triangulation” does not triangulate evidence, and mentioning a “robustness check” does not mean one was performed.
AI Can Generate Explanations for Decisions That Were Never Actually Made That Way
Research manuscripts often require authors to justify choices: why this sample, why this theoretical framework, why this analytical method, why these variables, why this exclusion criterion?
If the researcher made the choice first and asks AI afterward to produce a persuasive rationale, the resulting explanation may create a false impression that the decision emerged from the reasoning described.
Sometimes retrospective justification is legitimate. Researchers routinely articulate reasons more clearly during writing than they did in real time. The problem appears when AI manufactures a rationale that was not actually part of the methodological logic and that the researcher cannot substantiate independently.
A plausible justification is not evidence that the decision was justified.
AI Can Make Unsupported Interpretations Sound Inevitable
Research findings are often compatible with multiple explanations.
Generative AI is very good at selecting one possibility and turning it into a coherent narrative. Given a correlation, it can generate a mechanism. Given interview excerpts, it can produce a theme. Given an unexpected statistical result, it can supply a technical explanation.
Coherence can be persuasive because humans naturally respond to well-formed explanations. Yet an explanation may be coherent without being established by the data.
This becomes especially dangerous when AI-generated interpretations use technical terminology that gives the explanation an appearance of analytical authority.
The appropriate response is the same principle used when deciding which research tasks should not be delegated completely to AI: generated interpretations should become hypotheses for evaluation, not findings by linguistic default.
AI Can Produce Precision Without Evidential Precision
Specific numbers, dates, names, citations, formulas, methodological details, and technical language make writing feel authoritative.
Generative AI can produce all of them.
NIST identifies confabulation as a risk of generative AI: systems can confidently generate erroneous content, including false logic and fabricated citations. This is particularly consequential in domains requiring contextual or specialist knowledge.
A fabricated reference with authors, year, title, journal, volume, pages, and a DOI-like identifier may look more credible than a vague claim precisely because it is so detailed.
The detail is generated precision. It is not evidential verification.
AI Can Make a Manuscript Look More Complete Than the Research Process Was
A thin discussion section can be expanded. A sparse limitations section can become six paragraphs. A weak theoretical framework can acquire connections to several concepts. A brief methodological explanation can become impressively elaborate.
This creates what might be called synthetic completeness: missing intellectual work is replaced by generated text that resembles the product of that work.
The distinction becomes visible when you ask where each part came from.
| What the manuscript says |
What actual rigor would require |
What AI can imitate in prose |
| “We considered alternative explanations” |
Actual examination of plausible competing explanations |
A paragraph listing alternatives |
| “Robustness was assessed” |
Specified and executed robustness procedures |
A sophisticated description of robustness |
| “Themes were developed iteratively” |
An analytical process showing how themes were constructed and revised |
A plausible narrative of iterative analysis |
| “The instrument demonstrated validity” |
Appropriate evidence supporting the intended interpretation and use |
Technical language about validity |
| “Findings were triangulated” |
Actual comparison across appropriate sources, methods, investigators, or perspectives |
A paragraph describing the value of triangulation |
The issue is not the wording itself. Every statement above can accurately describe rigorous research. The problem is treating generated description as a substitute for the underlying procedure.
AI Can Produce Limitations Without Producing Methodological Self-Correction
Generative AI is usually capable of generating a respectable limitations section for almost any study.
That can be genuinely useful. Researchers sometimes overlook limitations, and an AI-generated list may expose issues that deserve attention.
But acknowledging a limitation after the study is completed is not equivalent to correcting a preventable problem during design.
If AI points out before data collection that a sampling strategy will systematically exclude an important population, the researcher may be able to improve the design. If the same issue appears only as polished limitation prose after data collection, the manuscript has become more self-aware, but the underlying evidence has not improved.
AI Can Make Citation Density Look Like Scholarly Depth
More citations do not necessarily mean better scholarship.
A generative system connected to retrieval tools may make it easy to insert numerous sources around an argument. Even when every citation is real, the researcher still needs to determine whether each source is relevant, accurately represented, methodologically credible, and genuinely supports the claim.
Without that evaluation, a paragraph can become citation-rich but intellectually shallow.
The problem is even greater when references are generated without reliable retrieval. A fabricated citation can provide the visual appearance of evidential support while contributing no evidence at all.
AI Can Conceal the Researcher's Evaluation Gap
A researcher can now produce technical material in areas where they have limited expertise.
That can be educationally useful. AI may explain unfamiliar code, statistical methods, or concepts and help researchers develop new skills.
It can also conceal an evaluation gap.
A manuscript may contain sophisticated machine-learning code, advanced statistical interpretation, or theoretical language that the named researchers cannot adequately explain or defend. Because the output is fluent and technically styled, the gap between production and understanding may not be visible to a reader.
This is one reason meaningful human oversight requires more than approving an output. Researchers need sufficient competence to evaluate consequential work produced with AI.
Polish Can Affect How Humans Judge Credibility
Readers do not evaluate research using methods alone. Presentation matters.
Clear writing can legitimately make good research easier to evaluate. Poor writing can obscure sound scholarship. Improving communication is therefore valuable.
But clarity, confidence, detail, and professional formatting can also increase perceived credibility independently of underlying evidential strength. Generative AI makes these signals inexpensive to produce at scale.
That means reviewers, supervisors, editors, and researchers themselves may need to become more deliberate about separating questions such as “Is this explanation convincing?” from “What evidence makes this explanation correct?”
The Problem Is Not That AI Makes Writing Better
Researchers should not respond by treating polished AI-assisted prose as suspicious merely because it is polished.
UNESCO explicitly discusses potential research uses of generative AI while emphasizing human agency, validation, privacy, and accountability. Its guidance notes potential for GenAI to expand perspectives on research outlines, data exploration, and literature reviews, while calling for evidence about efficacy and accuracy.
The European Commission likewise supports responsible adoption of generative AI while emphasizing research integrity, accountability, transparency, and responsibility. Its 2026 revision updates these safeguards as the technology changes.
The objective is therefore not to preserve bad writing as evidence of authenticity. It is to ensure that improved presentation does not outrun the quality of the research being presented.
A Useful Test Is to Remove the Prose Mentally
When evaluating an AI-polished study, imagine stripping away the methodological vocabulary and elegant explanation.
What remains?
- Was the research question actually answerable?
- Was the sample appropriate?
- Were the measures defensible?
- Were the procedures actually performed?
- Was the analysis suitable?
- Can the results be reproduced or audited where appropriate?
- Do the cited sources exist and support the claims?
- Do the conclusions follow from the evidence?
If those foundations are weak, generative AI may improve the manuscript without improving the study.
