Technical Report on the Pangram AI-Generated Text Classifier

2024/02/21 by Bradley Emi, Max Spero, Emi, Bradley +1 · 7 voices · 11 citations
Computer Science · #Text and Document Classification Technologies

paper · pdf · doi:10.48550/arxiv.2402.14873

Abstract

We present Pangram Text, a transformer-based neural network trained to distinguish text written by large language models from text written by humans. Pangram Text outperforms zero-shot methods such as DetectGPT as well as leading commercial AI detection tools with over 38 times lower error rates on a comprehensive benchmark comprised of 10 text domains (student writing, creative writing, scientific writing, books, encyclopedias, news, email, scientific papers, short-form Q&A) and 8 open- and closed-source large language models. We propose a training algorithm, hard negative mining with synthetic mirrors, that enables our classifier to achieve orders of magnitude lower false positive rates on high-data domains such as reviews. Finally, we show that Pangram Text is not biased against nonnative English speakers and generalizes to domains and models unseen during training.

Cited by

• Team DACTYL at PAN 2026: Bayesian Data Mixing and Empirical X-risk Minimization for AI-text Detection
• Will the widespread use of large language models in scientific writing undermine scientists’ critical thinking?
• StoryScope: Investigating idiosyncrasies in AI fiction
• AI use in American newspapers is widespread, uneven, and rarely disclosed
• EditLens: Quantifying the Extent of AI Editing in Text
• People who frequently use ChatGPT for writing tasks are accurate and robust detectors of AI-generated text
• LLM-Generated or Human-Written? Comparing Review and Non-Review Papers on ArXiv
• 'Your AI Text is not Mine': Redefining and Evaluating AI-generated Text Detection under Realistic Assumptions
• Flipping the Dialogue: Training and Evaluating User Language Models
• Death of the Novel(ty): Beyond n-Gram Novelty as a Metric for Textual Creativity
• Synthetic Sources?: Auditing Generative Search Engine Citations for Evidence of AI-Generated Sources

Discussions

• AI tool detects LLM-generated text in research papers and peer reviews [lemmy, 34 points, 7 comments]
• Yes, to Pangram's credit they do explain how they work: arxiv.org/abs/2402.14873 [bsky, 6 points, 0 comments]
• Next: Daniel Evanko with 'Quantifying and Assessing the Use of Generative AI by Authors and Reviewers in the Cancer Research Field' We use Pangram text classifier arxiv.org/abs/2402.14873 We studied 4 [bsky, 5 points, 1 comments]
• Does anyone know how Pangram (commercial AI detector) works? Feel like people have started posting Pangram outputs all the time. ah they put out a whitepaper. it's a transformer-based classifier. they [bsky, 3 points, 1 comments]
• i'm reading their technical report but "we simply trained the AI on GPT-generated text" feels too easy for the level of accuracy they seem to have - not just in test cases, but in the real world arxiv [bsky, 2 points, 1 comments]
• https://bsky.app/profile/ruoshuiresearch.bsky.social/post/3mnmwhhb7fc2q [bsky, 1 points, 0 comments]
• Technical Report on the Pangram AI-Generated Text Classifier (2024) [hn, 1 points, 0 comments]

Related

• Pangram 4 Technical Report
• AI-generated Text Detection: A Multifaceted Approach to Binary and Multiclass Classification
• Detecting AI Generated Text Based on NLP and Machine Learning Approaches
• AI Generated Text Detection Using Instruction Fine-tuned Large Language and Transformer-Based Models
• Mast Kalandar at SemEval-2024 Task 8: On the Trail of Textual Origins: RoBERTa-BiLSTM Approach to Detect AI-Generated Text