Efficient Large Scale Language Modeling with Mixtures of Experts

2021/12/20 by Mikel Artetxe, Artetxe, Mikel, Shruti Bhosale +47 · 1 voice · 31 citations
Computer Science · Engineering · Mathematics · #Algorithm #Artificial intelligence #Autoregressive model #Code (set theory) #Computation #Computer science #Econometrics #Engineering #Expert finding and Q&A systems #Language model #Machine Learning in Healthcare #Mathematics #Programming language #Range (aeronautics) #Scale (ratio) #Scaling #Topic Modeling #cs.AI #cs.CL #cs.LG

paper · pdf · doi:10.48550/arxiv.2112.10684

published in arXiv (Cornell University) (Cornell University) · EMNLP 2022

openalex publication_date 2021/12/20 · arxiv created 2022/10/26 · arxiv updated 2022/10/27 · openalex created_date 2025/10/10 · openalex updated_date 2026/08/05

Abstract

Mixture of Experts layers (MoEs) enable efficient scaling of language models through conditional computation. This paper presents a detailed empirical study of how autoregressive MoE language models scale in comparison with dense models in a wide range of settings: in- and out-of-domain language modeling, zero- and few-shot priming, and full-shot fine-tuning. With the exception of fine-tuning, we find MoEs to be substantially more compute efficient. At more modest training budgets, MoEs can match the performance of dense models using ∼4 times less compute. This gap narrows at scale, but our largest MoE model (1.1T parameters) consistently outperforms a compute-equivalent dense model (6.7B parameters). Overall, this performance gap varies greatly across tasks and domains, suggesting that MoE and dense models generalize differently in ways that are worthy of future study. We make our code and models publicly available for research use.

Cited by

• Is MoE Routing a Huffman Code? Discovering the Frequency-Diversity Law in Chain-of-Thought
• MoE Pathfinder: Trajectory-driven Expert Pruning
• Beyond Real Weights: Hypercomplex Representations for Stable Quantization
• Adaptive Defense against Harmful Fine-Tuning for Large Language Models via Bayesian Data Scheduler
• GatePro: Parameter-Free Expert Selection Optimization for Mixture-of-Experts Models
• Sparsely gated tiny linear experts
• Multilingual Routing in Mixture-of-Experts
• Skip-It? Theoretical Conditions for Layer Skipping in Vision-Language Models
• Towards Cross-Task Suicide Risk Detection via Speech LLM
• Combating the Memory Walls: Optimization Pathways for Long-Context Agentic LLM Inference
• Ban&Pick: Ehancing Performance and Efficiency of MoE-LLMs via Smarter Routing
• Dynamic Adaptive Shared Experts with Grouped Multi-Head Attention Mixture of Experts
• HAP: Hybrid Adaptive Parallelism for Efficient Mixture-of-Experts Inference
• Is GPT-OSS Good? A Comprehensive Evaluation of OpenAI's Latest Open Source Models
• What are the limits to biomedical research acceleration through general-purpose AI?
• EAC-MoE: Expert-Selection Aware Compressor for Mixture-of-Experts Large Language Models
• Contextual feature extraction hierarchies converge in large language models and the brain
• Mixture of Experts in Large Language Models
• Data Efficacy for Language Model Training
• MoORE: SVD-based Model MoE-ization for Conflict- and Oblivion-Resistant Multi-Task Adaptation
• Statement-Tuning Enables Efficient Cross-lingual Generalization in Encoder-only Models
• Retrieval-Augmented Generation Systems for Intellectual Property via Synthetic Multi-Angle Fine-tuning
• Advancing Expert Specialization for Better MoE
• Curse of High Dimensionality Issue in Transformer for Long-context Modeling
• BadPatches: Backdoor Attacks Against Patch-based Mixture of Experts Architectures
• Improving Routing in Sparse Mixture of Experts with Graph of Tokens
• Mixture of Experts for Decentralized Generative AI and Reinforcement Learning in Wireless Networks: A Comprehensive Survey
• Revisiting Transformers through the Lens of Low Entropy and Dynamic Sparsity
• Cross-region Model Training with Communication-Computation Overlapping and Delay Compensation
• MoE Parallel Folding: Heterogeneous Parallelism Mappings for Efficient Large-Scale MoE Model Training with Megatron Core
• MiLo: Efficient Quantized MoE Inference with Mixture of Low-Rank Compensators

Discussions

• Efficient Large Scale Language Modeling with Mixtures of Experts [hn, 25 points, 0 comments]

Related

• Scaling Fine-Grained MoE Beyond 50B Parameters: Empirical Evaluation and Practical Insights
• Scaling Laws for Fine-Grained Mixture of Experts
• Towards Greater Leverage: Scaling Laws for Efficient Mixture-of-Experts Language Models
• Mixture of Experts in Large Language Models
• Toward Inference-optimal Mixture-of-Expert Large Language Models