LLaMA-MoE: Building Mixture-of-Experts from LLaMA with Continual Pre-training

2024/06/24 by Tong Zhu, Xiaoye Qu, Zhu, Tong +11 · 35 citations
Computer Science · Engineering · #Computation and Language (cs.CL) #Context-Aware Activity Recognition Systems #FOS: Computer and information sciences #Indoor and Outdoor Localization Technologies #Mobile Crowdsensing and Crowdsourcing

paper · pdf · doi:10.48550/arxiv.2406.16554

openalex publication_date 2024/06/24 · openalex created_date 2024/06/26 · openalex updated_date 2026/07/28

Abstract

Mixture-of-Experts (MoE) has gained increasing popularity as a promising framework for scaling up large language models (LLMs). However, training MoE from scratch in a large-scale setting still suffers from data-hungry and instability problems. Motivated by this limit, we investigate building MoE models from existing dense large language models. Specifically, based on the well-known LLaMA-2 7B model, we obtain an MoE model by: (1) Expert Construction, which partitions the parameters of original Feed-Forward Networks (FFNs) into multiple experts; (2) Continual Pre-training, which further trains the transformed MoE model and additional gate networks. In this paper, we comprehensively explore different methods for expert construction and various data sampling strategies for continual pre-training. After these stages, our LLaMA-MoE models could maintain language abilities and route the input tokens to specific experts with part of the parameters activated. Empirically, by training 200B tokens, LLaMA-MoE-3.5B models significantly outperform dense models that contain similar activation parameters. The source codes and models are available at https://github.com/pjlab-sys4nlp/llama-moe .

Cited by

• How Many Experts Are Enough? Towards Optimal Semantic Specialization for Mixture-of-Experts
• PoseMoE: Mixture-of-Experts Network for Monocular 3D Human Pose Estimation
• What really matters for person re-identification? A Mixture-of-Experts Framework for Semantic Attribute Importance
• HiMoE-VLA: Hierarchical Mixture-of-Experts for Generalist Vision-Language-Action Policies
• ExpertAD: Enhancing Autonomous Driving Systems with Mixture of Experts
• MoRE: 3D Visual Geometry Reconstruction Meets Mixture-of-Experts
• MIN-Merging: Merge the Important Neurons for Model Merging
• MoE-Prism: Disentangling Monolithic Experts for Elastic MoE Services via Model-System Co-Designs
• ReXMoE: Reusing Experts with Minimal Overhead in Mixture-of-Experts
• GatePro: Parameter-Free Expert Selection Optimization for Mixture-of-Experts Models
• Understanding the Mixture-of-Experts with Nadaraya-Watson Kernel
• Symphony-MoE: Harmonizing Disparate Pre-trained Models into a Coherent Mixture-of-Experts
• Too Helpful, Too Harmless, Too Honest or Just Right?
• Joint Learning using Mixture-of-Expert-Based Representation for Enhanced Speech Generation and Robust Emotion Recognition
• TRACE: Unlocking Effective CXL Bandwidth via Lossless Compression and Precision Scaling
• Cache Management for Mixture-of-Experts LLMs -- extended version
• Federated Fine-Tuning of Sparsely-Activated Large Language Models on Resource-Constrained Devices
• RailX: A Flexible, Scalable, and Low-Cost Network Architecture for Hyper-Scale LLM Training Systems
• UniPool: A Globally Shared Expert Pool for Mixture-of-Experts
• DIVE into MoE: Diversity-Enhanced Reconstruction of Large Language Models from Dense into Mixture-of-Experts
• SlimCaching: Edge Caching of Mixture-of-Experts for Distributed Inference
• What You Have is What You Track: Adaptive and Robust Multimodal Tracking
• MoE-GPS: Guidlines for Prediction Strategy for Dynamic Expert Duplication in MoE Load Balancing
• Scatter-Based Innovation Propagation in Large Language Models for Multi-Stage Process Adaptation
• EAQuant: Enhancing Post-Training Quantization for MoE Models via Expert-Aware Optimization
• Mixture-of-Experts Meets In-Context Reinforcement Learning
• SkipGPT: Dynamic Layer Pruning Reinvented with Token Awareness and Module Decoupling
• Advancing Decoding Strategies: Enhancements in Locally Typical Sampling for LLMs
• Mixture-of-Experts for Personalized and Semantic-Aware Next Location Prediction
• Advancing Expert Specialization for Better MoE
• EvoMoE: Expert Evolution in Mixture of Experts for Multimodal Large Language Models
• Uni3D-MoE: Scalable Multimodal 3D Scene Understanding via Mixture of Experts
• DriveMoE: Mixture-of-Experts for Vision-Language-Action Model in End-to-End Autonomous Driving
• Not All Models Suit Expert Offloading: On Local Routing Consistency of Mixture-of-Expert Models
• Balanced and Elastic End-to-end Training of Dynamic LLMs

Related

• Llama 3 Meets MoE: Efficient Upcycling
• LLaMA-MoE v2: Exploring Sparsity of LLaMA from Perspective of Mixture-of-Experts with Post-Training
• dots.llm1 Technical Report
• L-MoE: End-to-End Training of a Lightweight Mixture of Low-Rank Adaptation Experts
• Can Mixture-of-Experts Surpass Dense LLMs Under Strictly Equal Resources?