TinyBERT: Distilling BERT for Natural Language Understanding

2019/09/23 by Jiao, Xiaoqi, Yin, Yichun, Shang, Lifeng +5 · 80 citations
#Artificial Intelligence (cs.AI) #Computation and Language (cs.CL) #FOS: Computer and information sciences #Machine Learning (cs.LG)

paper · doi:10.48550/arxiv.1909.10351

Abstract

Language model pre-training, such as BERT, has significantly improved the performances of many natural language processing tasks. However, pre-trained language models are usually computationally expensive, so it is difficult to efficiently execute them on resource-restricted devices. To accelerate inference and reduce model size while maintaining accuracy, we first propose a novel Transformer distillation method that is specially designed for knowledge distillation (KD) of the Transformer-based models. By leveraging this new KD method, the plenty of knowledge encoded in a large teacher BERT can be effectively transferred to a small student Tiny-BERT. Then, we introduce a new two-stage learning framework for TinyBERT, which performs Transformer distillation at both the pretraining and task-specific learning stages. This framework ensures that TinyBERT can capture he general-domain as well as the task-specific knowledge in BERT. TinyBERT with 4 layers is empirically effective and achieves more than 96.8% the performance of its teacher BERTBASE on GLUE benchmark, while being 7.5x smaller and 9.4x faster on inference. TinyBERT with 4 layers is also significantly better than 4-layer state-of-the-art baselines on BERT distillation, with only about 28% parameters and about 31% inference time of them. Moreover, TinyBERT with 6 layers performs on-par with its teacher BERTBASE.

Cited by

• LLMCache: Layer-Wise Caching Strategies for Accelerated Reuse in Transformer Inference
• EdgeFlex-Transformer: Transformer Inference for Edge Devices
• TiME: Tiny Monolingual Encoders for Efficient NLP Pipelines
• Beyond Real Weights: Hypercomplex Representations for Stable Quantization
• Art2Music: Generating Music for Art Images with Multi-modal Feeling Alignment
• Odin: Oriented Dual-module Integration for Text-rich Network Representation Learning
• Towards Edge General Intelligence: Knowledge Distillation for Mobile Agentic AI
• Deterministic Continuous Replacement: Fast and Stable Module Replacement in Pretrained Transformers
• A Systematic Study of Compression Ordering for Large Language Models
• NX-CGRA: A Programmable Hardware Accelerator for Core Transformer Algorithms on Edge Devices
• When Structure Doesn't Help: LLMs Do Not Read Text-Attributed Graphs as Effectively as We Expected
• Unifying points of interest taxonomies: mapping OpenStreetMap tags to the Foursquare category system
• Dynamic Temperature Scheduler for Knowledge Distillation
• Enhancing the Medical Context-Awareness Ability of LLMs via Multifaceted Self-Refinement Learning
• Black-Box On-Policy Distillation of Large Language Models
• CleverBirds: A Multiple-Choice Benchmark for Fine-grained Human Knowledge Tracing
• Self-Correction Distillation for Structured Data Question Answering
• Two Heads are Better than One: Distilling Large Language Model Features Into Small Models with Feature Decomposition and Mixture
• MobileLLM-Pro Technical Report
• CAMP-HiVe: Cyclic Pair Merging based Efficient DNN Pruning with Hessian-Vector Approximation for Resource-Constrained Systems
• A Metamorphic Testing Perspective on Knowledge Distillation for Language Models of Code: Does the Student Deeply Mimic the Teacher?
• Thinking with DistilQwen: A Tale of Four Distilled Reasoning and Reward Model Series
• Reviving Stale Updates: Data-Free Knowledge Distillation for Asynchronous Federated Learning
• Elastic Architecture Search for Efficient Language Models
• Distilling Multilingual Vision-Language Models: When Smaller Models Stay Multilingual
• SPEAR: A Unified SSL Framework for Learning Speech and Audio Representations
• FakeZero: Real-Time, Privacy-Preserving Misinformation Detection for Facebook and X
• FHECore: Rethinking GPU Microarchitecture for Fully Homomorphic Encryption
• SwiftEmbed: Ultra-Fast Text Embeddings via Static Token Lookup for Real-Time Applications
• FastVLM: Self-Speculative Decoding for Fast Vision-Language Model Inference
• Few-Shot Knowledge Distillation of LLMs With Counterfactual Explanations
• SindBERT, the Sailor: Charting the Seas of Turkish NLP
• TernaryCLIP: Efficiently Compressing Vision-Language Models with Ternary Weights and Distilled Knowledge
• Amplifying Prominent Representations in Multimodal Learning via Variational Dirichlet Process
• Mixture of Experts Approaches in Dense Retrieval Tasks
• Efficient Adaptive Transformer: An Empirical Study and Reproducible Framework
• CompoDistill: Attention Distillation for Compositional Reasoning in Multimodal LLMs
• A Survey on Collaborating Small and Large Language Models for Performance, Cost-effectiveness, Cloud-edge Privacy, and Trustworthiness
• Where to Begin: Efficient Pretraining via Subnetwork Selection and Distillation
• GUIDE: Guided Initialization and Distillation of Embeddings
• Downsized and Compromised?: Assessing the Faithfulness of Model Compression
• Learning from All: Concept Alignment for Autonomous Distillation from Multiple Drifting MLLMs
• Layer-wise dynamic rank for compressing large language models
• CURA: Size Isnt All You Need -- A Compact Universal Architecture for On-Device Intelligence
• Knowledge distillation through geometry-aware representational alignment
• RestoRect: Degraded Image Restoration via Latent Rectified Flow & Feature Distillation
• Progressive Weight Loading: Accelerating Initial Inference and Gradually Boosting Performance on Resource-Constrained Environments
• COSPADI: Compressing LLMs via Calibration-Guided Sparse Dictionary Learning
• MonoCon: A general framework for learning ultra-compact high-fidelity representations using monotonicity constraints
• Otters: An Energy-Efficient SpikingTransformer via Optical Time-to-First-Spike Encoding
• Flatness is Necessary, Neural Collapse is Not: Rethinking Generalization via Grokking
• LLM on a Budget: Active Knowledge Distillation for Efficient Classification of Large Text Corpora
• LEAF: Knowledge Distillation of Text Embedding Models with Teacher-Aligned Representations
• Spatio-Temporal Pruning for Compressed Spiking Large Language Models
• A Transformer-Based Cross-Platform Analysis of Public Discourse on the 15-Minute City Paradigm
• AMMKD: Adaptive Multimodal Multi-teacher Distillation for Lightweight Vision-Language Models
• Mitigating Attention Localization in Small Scale: Self-Attention Refinement via One-step Belief Propagation
• NoteBar: An AI-Assisted Note-Taking System for Personal Knowledge Management
• A Multimodal-Multitask Framework with Cross-modal Relation and Hierarchical Interactive Attention for Semantic Comprehension
• Expandable Residual Approximation for Knowledge Distillation
• Efficient Large Language Models with Zero-Shot Adjustable Acceleration
• SLM-Bench: A Comprehensive Benchmark of Small Language Models on Environmental Impacts--Extended Version
• An Empirical Study of Knowledge Distillation for Code Understanding Tasks
• Checkmate: interpretable and explainable RSVQA is the endgame
• Recent Advances in Transformer and Large Language Models for UAV Applications
• Computational Economics in Large Language Models: Exploring Model Behavior and Incentive Design under Resource Constraints
• Personalized Product Search Ranking: A Multi-Task Learning Approach with Tabular and Non-Tabular Data
• Bhav-Net: Knowledge Transfer for Cross-Lingual Antonym vs Synonym Distinction via Dual-Space Graph Transformers
• GeRe: Towards Efficient Anti-Forgetting in Continual Learning of LLM via General Samples Replay
• Model Compression vs. Adversarial Robustness: An Empirical Study on Language Models for Code
• Distillation-Enhanced Clustering Acceleration for Encrypted Traffic Classification
• DACTYL: Diverse Adversarial Corpus of Texts Yielded from Large Language Models
• Enhanced Arabic Text Retrieval with Attentive Relevance Scoring
• On the Sustainability of AI Inferences in the Edge
• Model-free Speculative Decoding for Transformer-based ASR with Token Map Drafting
• Investigating Structural Pruning and Recovery Techniques for Compressing Multimodal Large Language Models: An Empirical Study
• The Carbon Cost of Conversation, Sustainability in the Age of Language Models
• Basic Reading Distillation
• Breaking the Tokenizer Barrier: On-Policy Distillation across Model Families
• OPRD: On-Policy Representation Distillation

Related

• DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter
• ERNIE-Tiny : A Progressive Distillation Framework for Pretrained Transformer Compression
• Improving Knowledge Distillation for BERT Models: Loss Functions, Mapping Methods, and Weight Tuning
• RefBERT: Compressing BERT by Referencing to Pre-computed Representations
• Distilling BERT into Simple Neural Networks with Unlabeled Transfer Data