2020/05/02 by Tal Linzen, Linzen, Tal · 3 citations
Computer Science · #Computation and Language (cs.CL) #FOS: Computer and information sciences #Natural Language Processing Techniques #Speech Recognition and Synthesis #Topic Modeling
paper · pdf · doi:10.48550/arxiv.2005.00955
openalex publication_date 2020/05/02 · openalex created_date 2022/07/26 · openalex updated_date 2026/07/28
This position paper describes and critiques the Pretraining-Agnostic\nIdentically Distributed (PAID) evaluation paradigm, which has become a central\ntool for measuring progress in natural language understanding. This paradigm\nconsists of three stages: (1) pre-training of a word prediction model on a\ncorpus of arbitrary size; (2) fine-tuning (transfer learning) on a training set\nrepresenting a classification task; (3) evaluation on a test set drawn from the\nsame distribution as that training set. This paradigm favors simple, low-bias\narchitectures, which, first, can be scaled to process vast amounts of data, and\nsecond, can capture the fine-grained statistical properties of a particular\ndata set, regardless of whether those properties are likely to generalize to\nexamples of the task outside the data set. This contrasts with humans, who\nlearn language from several orders of magnitude less data than the systems\nfavored by this evaluation paradigm, and generalize to new tasks in a\nconsistent way. We advocate for supplementing or replacing PAID with paradigms\nthat reward architectures that generalize as quickly and robustly as humans.\n