ELLE: Efficient Lifelong Pre-training for Emerging Data

2022/03/12 by Yujia Qin, Qin, Yujia, Jiajie Zhang +11 · 9 citations
Computer Science · #Artificial Intelligence (cs.AI) #Computation and Language (cs.CL) #FOS: Computer and information sciences #cs.AI #cs.CL

paper · pdf · doi:10.48550/arxiv.2203.06311

Findings of ACL 2022

arxiv created 2022/07/11 · arxiv updated 2022/07/12

Abstract

Current pre-trained language models (PLM) are typically trained with static data, ignoring that in real-world scenarios, streaming data of various sources may continuously grow. This requires PLMs to integrate the information from all the sources in a lifelong manner. Although this goal could be achieved by exhaustive pre-training on all the existing data, such a process is known to be computationally expensive. To this end, we propose ELLE, aiming at efficient lifelong pre-training for emerging data. Specifically, ELLE consists of (1) function preserved model expansion, which flexibly expands an existing PLM's width and depth to improve the efficiency of knowledge acquisition; and (2) pre-trained domain prompts, which disentangle the versatile knowledge learned during pre-training and stimulate the proper knowledge for downstream tasks. We experiment ELLE with streaming data from 5 domains on BERT and GPT. The results show the superiority of ELLE over various lifelong learning baselines in both pre-training efficiency and downstream performances. The codes are publicly available at https://github.com/thunlp/ELLE.

Cited by

• Mitigating Forgetting in Low Rank Adaptation
• AnaCP: Toward Upper-Bound Continual Learning via Analytic Contrastive Projection
• Deep Progressive Training: scaling up depth capacity of zero/one-layer models
• SCALE: Upscaled Continual Learning of Large Language Models
• Robust Uncertainty Quantification for Self-Evolving Large Language Models via Continual Domain Pretraining
• Predicting Training Re-evaluation Curves Enables Effective Data Curriculums for LLMs
• Self-Evolving LLMs via Continual Instruction Tuning
• UltraEdit: Training-, Subject-, and Memory-Free Lifelong Editing in Language Models
• Memorization vs. Reasoning: Updating LLMs with New Knowledge

Related

• Lifelong Pretraining: Continually Adapting Language Models to Emerging Corpora
• INGENIOUS: Using Informative Data Subsets for Efficient Pre-Training of Language Models
• Lifelong Language Pretraining with Distribution-Specialized Experts
• Language Models as Continuous Self-Evolving Data Engineers
• Thinking Augmented Pre-training