Elastic Architecture Search for Efficient Language Models

2025/10/30 by Wang, Shang
Computer Science · #Artificial Intelligence (cs.AI) #Computation and Language (cs.CL) #FOS: Computer and information sciences #Machine Learning (cs.LG) #Multimodal Machine Learning Applications #Natural Language Processing Techniques #Neural and Evolutionary Computing (cs.NE) #Topic Modeling

paper · doi:10.48550/arxiv.2510.27037

openalex publication_date 2025/10/30 · openalex created_date 2025/11/05 · openalex updated_date 2026/07/28

Abstract

As large pre-trained language models become increasingly critical to natural language understanding (NLU) tasks, their substantial computational and memory requirements have raised significant economic and environmental concerns. Addressing these challenges, this paper introduces the Elastic Language Model (ELM), a novel neural architecture search (NAS) method optimized for compact language models. ELM extends existing NAS approaches by introducing a flexible search space with efficient transformer blocks and dynamic modules for dimension and head number adjustment. These innovations enhance the efficiency and flexibility of the search process, which facilitates more thorough and effective exploration of model architectures. We also introduce novel knowledge distillation losses that preserve the unique characteristics of each block, in order to improve the discrimination between architectural choices during the search process. Experiments on masked language modeling and causal language modeling tasks demonstrate that models discovered by ELM significantly outperform existing methods.

Citations

• LLaMA: Open and Efficient Foundation Language Models
• Self-Instruct: Aligning Language Models with Self-Generated Instructions
• Unnatural Instructions: Tuning Language Models with (Almost) No Human Labor
• PKD: General Distillation Framework for Object Detectors via Pearson Correlation Coefficient
• Knowledge Distillation from A Stronger Teacher
• EfficientBERT: Progressively Searching Multilayer Perceptron via Warm-up Knowledge Distillation
• AutoBERT-Zero: Evolving BERT Backbone from Scratch
• MobileBERT: a Compact Task-Agnostic BERT for Resource-Limited Devices
• MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformers
• AdaBERT: Task-Adaptive BERT Compression with Differentiable Neural Architecture Search
• DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter
• TinyBERT: Distilling BERT for Natural Language Understanding
• Patient Knowledge Distillation for BERT Model Compression
• RoBERTa: A Robustly Optimized BERT Pretraining Approach
• Similarity of Neural Network Representations Revisited
• Single Path One-Shot Neural Architecture Search with Uniform Sampling
• GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding
• SQuAD: 100,000+ Questions for Machine Comprehension of Text
• Aligning Books and Movies: Towards Story-like Visual Explanations by Watching Movies and Reading Books
• BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

Related

• Structural Pruning of Pre-trained Language Models via Neural Architecture Search
• Learning Architectures from an Extended Search Space for Language Modeling
• Neural Architecture Search for Parameter-Efficient Fine-tuning of Large Pre-trained Language Models
• NAS-Bench-NLP: Neural Architecture Search Benchmark for Natural Language Processing
• SEKI: Self-Evolution and Knowledge Inspiration based Neural Architecture Search via Large Language Models