FastBERT: a Self-distilling BERT with Adaptive Inference Time

2020/04/05 by Weijie Liu, Liu, Weijie, Peng Zhou +9 · 25 citations
Computer Science · #Computation and Language (cs.CL) #FOS: Computer and information sciences #Natural Language Processing Techniques #Speech Recognition and Synthesis #Topic Modeling #cs.CL

paper · pdf · doi:10.48550/arxiv.2004.02178

This manuscript has been accepted to appear at ACL 2020

openalex publication_date 2020/04/05 · arxiv created 2020/04/29 · arxiv updated 2020/04/30 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Pre-trained language models like BERT have proven to be highly performant. However, they are often computationally expensive in many practical scenarios, for such heavy models can hardly be readily implemented with limited resources. To improve their efficiency with an assured model performance, we propose a novel speed-tunable FastBERT with adaptive inference time. The speed at inference can be flexibly adjusted under varying demands, while redundant calculation of samples is avoided. Moreover, this model adopts a unique self-distillation mechanism at fine-tuning, further enabling a greater computational efficacy with minimal loss in performance. Our model achieves promising results in twelve English and Chinese datasets. It is able to speed up by a wide range from 1 to 12 times than BERT if given different speedup thresholds to make a speed-performance tradeoff.

Citations

• FitNets: Hints for Thin Deep Nets
• Compressing Deep Convolutional Networks using Vector Quantization
• Distilling the Knowledge in a Neural Network
• Adaptive Computation Time for Recurrent Neural Networks
• ERNIE: Enhanced Representation through Knowledge Integration
• XLNet: Generalized Autoregressive Pretraining for Language Understanding
• LLM-Supported Manufacturing Mapping Generation

Cited by

• DeeAD: Dynamic Early Exit of Vision-Language Action for Efficient Autonomous Driving
• HatePrototypes: Interpretable and Transferable Representations for Implicit and Explicit Hate Speech Detection
• Decoupled Multi-Predictor Optimization for Inference-Efficient Model Tuning
• Dynamic Parameterization Is Not Dynamic Inference
• Efficient Adaptive Transformer: An Empirical Study and Reproducible Framework
• Nav-EE: Navigation-Guided Early Exiting for Efficient Vision-Language Models in Autonomous Driving
• TIDE: Token-Informed Depth Execution for Per-Token Early Exit in LLM Inference
• Learning What to Remember: Adaptive Probabilistic Memory Retention for Memory-Efficient Language Models
• DynaNav: Dynamic Feature and Layer Selection for Efficient Visual Navigation
• Harnessing Input-Adaptive Inference for Efficient VLN
• Basic Reading Distillation
• GNN-CNN: An Efficient Hybrid Model of Convolutional and Graph Neural Networks for Text Representation
• Skip a Layer or Loop it? Test-Time Depth Adaptation of Pretrained LLMs
• Early Stopping Tabular In-Context Learning
• Improving Prediction Certainty Estimation for Reliable Early Exiting via Null Space Projection
• FREE: Fast and Robust Vision Language Models with Early Exits
• Dynamic Neural Networks: A Survey
• AD-EE: Early Exiting for Fast and Reliable Vision-Language Models in Autonomous Driving
• Knowledge Distillation: A Survey
• IOT: Instance-wise Layer Reordering for Transformer Structures
• Reroute, Don't Remove: Recoverable Visual Token Routing for Vision-Language Models
• DYNAMAX: Dynamic computing for Transformers and Mamba based architectures
• Improving NER's Performance with Massive financial corpus
• Learning Dynamic BERT via Trainable Gate Variables and a Bi-modal Regularizer
• MACRO: Markov Chain Routing of Transformer Layers

Related

• Elbert: Fast Albert with Confidence-Window Based Early Exit
• SmartBERT: A Promotion of Dynamic Early Exiting Mechanism for Accelerating BERT Inference
• DACT-BERT: Differentiable Adaptive Computation Time for an Efficient BERT Inference
• DeeBERT: Dynamic Early Exiting for Accelerating BERT Inference
• Improving BERT Fine-Tuning via Self-Ensemble and Self-Distillation