Undivided Attention: Are Intermediate Layers Necessary for BERT?

2020/12/22 by Sharath Nittur Sridhar, Sridhar, Sharath Nittur, Anthony Sarah +1 · 3 citations
Computer Science · #Artificial Intelligence (cs.AI) #Computation and Language (cs.CL) #FOS: Computer and information sciences #Multimodal Machine Learning Applications #Natural Language Processing Techniques #Topic Modeling

paper · pdf · doi:10.48550/arxiv.2012.11881

openalex publication_date 2020/12/22 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

In recent times, BERT-based models have been extremely successful in solving a variety of natural language processing (NLP) tasks such as reading comprehension, natural language inference, sentiment analysis, etc. All BERT-based architectures have a self-attention block followed by a block of intermediate layers as the basic building component. However, a strong justification for the inclusion of these intermediate layers remains missing in the literature. In this work we investigate the importance of intermediate layers on the overall network performance of downstream tasks. We show that reducing the number of intermediate layers and modifying the architecture for BERT-BASE results in minimal loss in fine-tuning accuracy for downstream tasks while decreasing the number of parameters and training time of the model. Additionally, we use centered kernel alignment and probing linear classifiers to gain insight into our architectural modifications and justify that removal of intermediate layers has little impact on the fine-tuned accuracy.

Citations

• Gaussian Error Linear Units (GELUs)
• Aligning Books and Movies: Towards Story-like Visual Explanations by Watching Movies and Reading Books
• Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism
• LLM-Supported Manufacturing Mapping Generation
• ALBERT: A Lite BERT for Self-supervised Learning of Language Representations
• Language Models are Few-Shot Learners

Cited by

• Fine-Tuned Transformers Show Clusters of Similar Representations Across Layers
• Attention Is Not All You Need: The Importance of Feedforward Networks in Transformer Models
• TrimBERT: Tailoring BERT for Trade-offs

Related

• TrimBERT: Tailoring BERT for Trade-offs
• Revealing the Dark Secrets of BERT
• LV-BERT: Exploiting Layer Variety for BERT
• CoRe: An Efficient Coarse-refined Training Framework for BERT
• Distilling BERT for low complexity network training