A Survey on Model Compression and Acceleration for Pretrained Language Models

2022/02/15 by Canwen Xu, Julian McAuley, Xu, Canwen +1 · 7 citations
Computer Science · #Artificial Intelligence (cs.AI) #Computation and Language (cs.CL) #FOS: Computer and information sciences #Machine Learning (cs.LG) #Machine Learning and Data Classification #Natural Language Processing Techniques #Topic Modeling

paper · pdf · doi:10.48550/arxiv.2202.07105

openalex publication_date 2022/02/15 · openalex created_date 2022/05/05 · openalex updated_date 2026/07/28

Abstract

Despite achieving state-of-the-art performance on many NLP tasks, the high energy cost and long inference delay prevent Transformer-based pretrained language models (PLMs) from seeing broader adoption including for edge and mobile computing. Efficient NLP research aims to comprehensively consider computation, time and carbon emission for the entire life-cycle of NLP, including data preparation, model training and inference. In this survey, we focus on the inference stage and review the current state of model compression and acceleration for pretrained language models, including benchmarks, metrics and methodology.

Cited by

• A Metamorphic Testing Perspective on Knowledge Distillation for Language Models of Code: Does the Student Deeply Mimic the Teacher?
• StructPrune: Structured Global Pruning asymptotics with O(√(N)) GPU Memory
• CLARIFY: A Specialist-Generalist Framework for Accurate and Lightweight Dermatological Visual Question Answering
• An Empirical Study of Knowledge Distillation for Code Understanding Tasks
• Provable Post-Training Quantization: Theoretical Analysis of OPTQ and Qronos
• Model Compression vs. Adversarial Robustness: An Empirical Study on Language Models for Code
• How Quantization Impacts Privacy Risk on LLMs for Code?

Related

• Compression Laws for Large Language Models
• Towards Sustainable NLP: Insights from Benchmarking Inference Energy in Large Language Models
• Variator: Accelerating Pre-trained Models with Plug-and-Play Compression Modules
• From Transformers to LLMs: A Systematic Survey of Efficiency Considerations in NLP
• The Cost of Compression: Investigating the Impact of Compression on Parametric Knowledge in Language Models