OLMoE: Open Mixture-of-Experts Language Models

2024/09/03 by Niklas Muennighoff, Luca Soldaini, Muennighoff, Niklas +45 · 1 voice · 72 citations
Computer Science · #Expert finding and Q&A systems #Natural Language Processing Techniques #Topic Modeling #cs.AI #cs.CL #cs.LG

paper · pdf · doi:10.48550/arxiv.2409.02060

openalex publication_date 2024/09/03 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

We introduce OLMoE, a fully open, state-of-the-art language model leveraging sparse Mixture-of-Experts (MoE). OLMoE-1B-7B has 7 billion (B) parameters but uses only 1B per input token. We pretrain it on 5 trillion tokens and further adapt it to create OLMoE-1B-7B-Instruct. Our models outperform all available models with similar active parameters, even surpassing larger ones like Llama2-13B-Chat and DeepSeekMoE-16B. We present various experiments on MoE training, analyze routing in our model showing high specialization, and open-source all aspects of our work: model weights, training data, code, and logs.

Cited by

• MoE²-LoRA: When MoE Models Meet MoE-style Low-Rank Adaptation
• Breaking the MoE LLM Trilemma: Dynamic Expert Clustering with Structured Compression
• Diagnosing Overhead in Dispatch Operations: Cross-architecture Observatory
• Training Foundation Models on a Full-Stack AMD Platform: Compute, Networking, and System Design
• Chain-of-Experts: Unlocking the Communication Power of Mixture-of-Experts Models
• OLMoTrace: Tracing Language Model Outputs Back to Trillions of Training Tokens
• Enough Coin Flips Can Make LLMs Act Bayesian
• s1: Simple test-time scaling
• Knowledge without Wisdom: Measuring Misalignment between LLMs and Intended Impact
• UCCL-EP: Portable Expert-Parallel Communication
• SonicMoE: Accelerating MoE with IO and Tile-aware Optimizations
• DTop-p MoE: Sparsity-Controlled Dynamic Top-p MoE for Foundation Model Pre-training
• Resting Neurons, Active Insights: Improving Input Sparsification for Large Language Models
• PCMind-2.1-Kaiyuan-2B Technical Report
• Us-vs-Them bias in Large Language Models
• On the Origin of Algorithmic Progress in AI
• Routing Manifold Alignment Improves Generalization of Mixture-of-Experts LLMs
• Route Experts by Sequence, not by Token
• Deep Progressive Training: scaling up depth capacity of zero/one-layer models
• CryptoMoE: Privacy-Preserving and Scalable Mixture of Experts Inference via Balanced Expert Routing
• Diffusion Language Models are Super Data Learners
• Routing Matters in MoE: Scaling Diffusion Transformers with Explicit Routing Guidance
• Rewiring Experts on the Fly:Continuous Rerouting for Better Online Adaptation in Mixture-of-Expert models
• GatePro: Parameter-Free Expert Selection Optimization for Mixture-of-Experts Models
• MoBiLE: Efficient Mixture-of-Experts Inference on Consumer GPU with Mixture of Big Little Experts
• MC#: Mixture Compressor for Mixture-of-Experts Large Models
• Sparsely gated tiny linear experts
• KORMo: Korean Open Reasoning Model for Everyone
• MeSH: Memory-as-State-Highways for Recursive Transformers
• Beyond Sunk Costs: Boosting LLM Pre-training Efficiency via Orthogonal Growth of Mixture-of-Experts
• Patterns behind Chaos: Forecasting Data Movement for Efficient Large-Scale MoE LLM Inference
• Stratum: System-Hardware Co-Design with Tiered Monolithic 3D-Stackable DRAM for Efficient MoE Serving
• Multilingual Routing in Mixture-of-Experts
• MoME: Mixture of Matryoshka Experts for Audio-Visual Speech Recognition
• Scaling with Collapse: Efficient and Predictable Training of LLM Families
• GRACE-MoE: Grouping and Replication with Locality-Aware Routing for Efficient Distributed MoE Inference
• Training Optimal Large Diffusion Language Models
• Dynamic Experts Search: Enhancing Reasoning in Mixture-of-Experts LLMs at Test Time
• Defending MoE LLMs against Harmful Fine-Tuning via Safety Routing Alignment
• Hallucination reduction with CASAL: Contrastive Activation Steering For Amortized Learning
• EMO: Pretraining Mixture of Experts for Emergent Modularity
• MoEs Are Stronger than You Think: Hyper-Parallel Inference Scaling with RoE
• MoE-Inference-Bench: Performance Evaluation of Mixture of Expert Large Language and Vision Models
• AsyMoE: Leveraging Modal Asymmetry for Enhanced Expert Specialization in Large Vision-Language Models
• Cosine-Similarity Routing with Semantic Anchors for Interpretable Mixture-of-Experts Language Models
• Accelerating Frontier MoE Training with 3D Integrated Optics
• MoPEQ: Mixture of Mixed Precision Quantized Experts
• LExI: Layer-Adaptive Active Experts for Efficient MoE Model Inference
• When Thinking Backfires: Mechanistic Insights Into Reasoning-Induced Misalignment
• Universal Properties of Activation Sparsity in Modern Large Language Models
• UltraMemV2: Memory Networks Scaling to 120B Parameters with Superior Long-Context Learning
• Optimal Sparsity of Mixture-of-Experts Language Models for Reasoning Tasks
• DualSparse-MoE: Coordinating Tensor/Neuron-Level Sparsity with Expert Partition and Reconstruction
• Maximum Score Routing For Mixture-of-Experts
• MoIIE: Mixture of Intra- and Inter-Modality Experts for Large Vision Language Models
• Generalizing Scaling Laws for Dense and Sparse Large Language Models
• Bridging Brains and Models: MoE-Based Functional Lesions for Simulating and Rehabilitating Aphasia
• When Truth Is Overridden: Uncovering the Internal Origins of Sycophancy in Large Language Models
• Parameter-Efficient Routed Fine-Tuning: Mixture-of-Experts Demands Mixture of Adaptation Modules
• EAC-MoE: Expert-Selection Aware Compressor for Mixture-of-Experts Large Language Models
• A3D-MoE: Acceleration of Large Language Models with Mixture of Experts via 3D Heterogeneous Integration
• UniPool: A Globally Shared Expert Pool for Mixture-of-Experts
• EXAONE 4.0: Unified Large Language Models Integrating Non-reasoning and Reasoning Modes
• FlexOlmo: Open Language Models for Flexible Data Use
• Intrinsic Fingerprint of LLMs: Continue Training is NOT All You Need to Steal A Model!
• MoNE: Replacing Redundant Experts with Lightweight Novices for Structured Pruning of MoE
• TPTT: Transforming Pretrained Transformers into Titans
• The Condition Number as a Scale-Invariant Proxy for Information Encoding in Neural Units
• FLAME: Towards Federated Fine-Tuning Large Language Models Through Adaptive SMoE
• Mix-of-Language-Experts Architecture for Multilingual Programming
• Utility-Driven Speculative Decoding for Mixture-of-Experts
• MoTE: Mixture of Ternary Experts for Memory-efficient Large Multimodal Models

Discussions

• Allen AI has released a fully open (weights, data, and code) mixture-of-experts model. It's super small (1B active parameters out of 7B total) but has good performance for its size. Paper: arxiv.org/a [bsky, 33 points, 2 comments]

Related

• OpenMoE: An Early Effort on Open Mixture-of-Experts Language Models
• LLaMA-MoE: Building Mixture-of-Experts from LLaMA with Continual Pre-training
• FLAME-MoE: A Transparent End-to-End Research Platform for Mixture-of-Experts Language Models
• dots.llm1 Technical Report
• Grove MoE: Towards Efficient and Superior MoE LLMs with Adjugate Experts