2024/02/21 by Bradley Emi, Max Spero, Emi, Bradley +1 · 7 voices · 15 citations
Computer Science · #Artificial intelligence #Classifier (UML) #Computer science #Information retrieval #Natural language processing #Text and Document Classification Technologies
paper · pdf · doi:10.48550/arxiv.2402.14873
published in arXiv (Cornell University) (Cornell University)
openalex publication_date 2024/02/21 · openalex created_date 2024/02/28 · openalex updated_date 2026/07/28
We present Pangram Text, a transformer-based neural network trained to distinguish text written by large language models from text written by humans. Pangram Text outperforms zero-shot methods such as DetectGPT as well as leading commercial AI detection tools with over 38 times lower error rates on a comprehensive benchmark comprised of 10 text domains (student writing, creative writing, scientific writing, books, encyclopedias, news, email, scientific papers, short-form Q&A) and 8 open- and closed-source large language models. We propose a training algorithm, hard negative mining with synthetic mirrors, that enables our classifier to achieve orders of magnitude lower false positive rates on high-data domains such as reviews. Finally, we show that Pangram Text is not biased against nonnative English speakers and generalizes to domains and models unseen during training.