Can depth-adaptive BERT perform better on binary classification tasks

2021/11/22 by Jing Fan, Xin Zhang, Fan, Jing +7
Computer Science · #Anomaly Detection Techniques and Applications #Computation and Language (cs.CL) #FOS: Computer and information sciences #Machine Learning and Data Classification #Natural Language Processing Techniques

paper · pdf · doi:10.48550/arxiv.2111.10951

openalex publication_date 2021/11/22 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

In light of the success of transferring language models into NLP tasks, we ask whether the full BERT model is always the best and does it exist a simple but effective method to find the winning ticket in state-of-the-art deep neural networks without complex calculations. We construct a series of BERT-based models with different size and compare their predictions on 8 binary classification tasks. The results show there truly exist smaller sub-networks performing better than the full model. Then we present a further study and propose a simple method to shrink BERT appropriately before fine-tuning. Some extended experiments indicate that our method could save time and storage overhead extraordinarily with little even no accuracy loss.

Related

• Lessons Learned from Applying off-the-shelf BERT: There is no Silver Bullet
• The Right Tool for the Job: Matching Model and Instance Complexities
• Distilling BERT for low complexity network training
• Distilling Task-Specific Knowledge from BERT into Simple Neural Networks
• Undivided Attention: Are Intermediate Layers Necessary for BERT?