03 · What You Need to Know
Why AI-Detection Percentages Cannot Establish Academic Authorship
What Does an AI-Detection Percentage Represent?
There is no universal definition of an AI-detection percentage.
Different systems may report a classification confidence score, an estimate of the proportion of text identified as AI-generated, or a proprietary indicator derived from statistical features.
These quantities are not interchangeable.
For example, Turnitin describes its AI writing percentage as the proportion of qualifying text in a submission that its model identifies as likely AI-generated or AI-generated and subsequently paraphrased, according to the product's applicable reporting categories.
That percentage is not presented as the probability that the author used AI. Nor is it a direct measurement of how much writing was actually produced by a language model.
Other detectors may use different definitions. Researchers must therefore consult the specific provider's documentation rather than assuming that all displayed percentages have the same meaning.
Why Is a Classification Score Different From a Probability of Authorship?
A detector may calculate a score indicating how strongly a passage resembles examples of AI-generated writing.
That score does not necessarily represent a calibrated probability that the passage was actually generated by AI.
Calibration concerns whether predictions assigned a particular probability correspond to observed frequencies in an appropriately defined population.
For example, if a well-calibrated system assigns a 70% probability to many comparable cases, approximately 70% of those cases should belong to the predicted category over repeated evaluation.
However, even a calibrated probability would not constitute certainty about an individual manuscript. It would remain conditional on the model, reference population, and available information.
A proprietary score labeled 70% may not even have that probabilistic interpretation.
Detector score
A numerical output generated according to a particular system's classification and reporting procedures.
Probability of actual AI use
A probabilistic claim about whether AI was involved in producing the text, requiring appropriate statistical justification and contextual information.
Confusing these quantities is one of the most consequential mistakes in interpreting AI-detection reports.
Why Can a High Score Still Be Wrong?
Most general-purpose detectors analyze characteristics of the finished text rather than observing its creation.
Human-written academic prose can contain predictable terminology, standardized methodological descriptions, and conventional sentence structures that resemble generated writing.
Consequently, a detector may assign a high AI-likelihood classification to text that was written entirely by a human.
Published evaluations have documented this problem. Weber-Wulff and colleagues examined 14 AI-text detection tools and reported important limitations in their accuracy and reliability.
Liang and colleagues demonstrated substantial false-positive disparities in a particular evaluation of essays written by non-native English speakers.
These findings support the existence of classification errors without establishing the current error rate of every commercial system.
The possibility of false-positive AI classifications means that a high score cannot independently establish machine generation.
Why Can a Low Score Also Be Misleading?
A detector may fail to identify text that was genuinely generated by AI.
Such a result is a false negative when the evaluation defines the text as AI-generated but the system classifies it as human-written.
Research has demonstrated that paraphrasing and other text transformations can reduce detection effectiveness under particular conditions.
Moreover, a detector may perform differently when evaluating text from a language model that was not adequately represented in its training or validation data.
A low score therefore cannot establish that a manuscript was written without AI assistance.
This is why passing an AI detector does not prove human authorship.
Why Do False-Positive Rates Matter More Than a Headline Accuracy Claim?
Overall accuracy measures the proportion of correct classifications across an evaluation dataset.
However, accuracy alone does not reveal how frequently human-written material is incorrectly flagged or how often AI-generated text escapes detection.
A detector might perform well on a balanced benchmark while producing numerous incorrect positive classifications in a population where AI-generated writing is relatively uncommon.
The false-positive rate is especially important when a positive result may trigger an allegation against a researcher.
Yet even a low false-positive rate does not establish that an individual flagged manuscript is AI-generated.
To interpret a positive result, the prevalence of AI-generated material in the relevant population must also be considered.
How Does the Base-Rate Problem Affect AI-Detection Scores?
The base-rate problem arises when people interpret a positive test result without considering how common the condition being tested actually is.
In AI detection, the relevant prevalence is the proportion of texts that genuinely belong to the AI-generated category under the evaluation's definition.
Consider a hypothetical detector with 90% sensitivity and 95% specificity.
If only 5% of manuscripts in a population are genuinely AI-generated, the detector will correctly flag many generated manuscripts but may also incorrectly flag a substantial number of human-written ones.
The calculation demonstrates that a detector can have seemingly strong sensitivity and specificity while producing a positive result that is correct less than half the time in a particular population.
These figures are illustrative, not estimates of any commercial detector's performance.
The result also does not mean that every manuscript assigned a high proprietary detection percentage has a 48.6% probability of being AI-generated. Positive predictive value concerns a defined group of positive classifications under specified conditions.
Why Does the Writing Population Affect Interpretation?
Detector performance may vary across academic disciplines, writing styles, text lengths, and linguistic backgrounds.
Liang and colleagues found that seven evaluated GPT detectors disproportionately misclassified a collection of essays written by non-native English speakers.
Their findings demonstrate that overall performance statistics may conceal unequal error rates across populations.
For research institutions, this raises questions about whether a detector validated on one type of writing can be applied fairly to manuscripts produced under different conditions.
It also highlights the need to examine language-related differences in AI-detection errors before using scores in consequential decisions.
Why Does Agreement Among Multiple Detectors Not Establish Proof?
Several detectors may classify the same manuscript as AI-generated. Their agreement may appear persuasive, but it does not automatically provide independent confirmation.
Detectors may share training data, statistical features, model architectures, or assumptions about generated writing.
If their errors are correlated, agreement may reflect a common limitation rather than independent evidence of authorship.
Combining detector results requires evidence that the combination improves performance under relevant conditions.
Simply counting how many systems flag a passage does not establish that the classification is correct.
Why Is a Score Insufficient to Establish Research Misconduct?
Even if AI assistance were independently confirmed, investigators would still need to determine whether the assistance violated an applicable rule.
Policies differ across universities, journals, publishers, and research activities.
Some forms of AI-assisted language editing may be permitted, while substantive drafting or analysis may require disclosure or be subject to restrictions.
A detector cannot independently establish which tools were used, whether the researcher disclosed them, or whether the relevant policy was violated.
Formal research misconduct also has specific definitions in some jurisdictions. For example, US Public Health Service regulations address fabrication, falsification, and plagiarism, rather than treating every undisclosed use of AI as research misconduct.
Therefore, a detection score cannot substitute for identifying the alleged conduct, establishing the relevant facts, and applying the governing policy.
What Evidence Is More Directly Relevant Than a Detection Score?
Evidence about the actual development of a manuscript may be more informative than a statistical classification.
Depending on the allegation, relevant records may include dated drafts, revision histories, research notes, authenticated AI interactions, and correspondence concerning manuscript preparation.
Such evidence is not automatically conclusive. Its authenticity, completeness, and relevance must be evaluated.
For example, a saved AI conversation may establish that a researcher requested generated text without proving that the output was incorporated into the manuscript.
Conversely, a document history may demonstrate extensive human revision without proving that no AI-generated material was used earlier.
The evaluation of documentary evidence of suspected undisclosed AI use should focus on what each record can genuinely establish.
