Llama 3 Meets MoE: Efficient Upcycling

2024/12/13 by Aditya Vavre, Vavre, Aditya, Ethan He +11 · 2 citations
Computer Science · Medicine · #Topic Modeling #Multimodal Machine Learning Applications #Artificial Intelligence in Healthcare and Education

paper · pdf · doi:10.48550/arxiv.2412.09952

Abstract

Scaling large language models (LLMs) significantly improves performance but\ncomes with prohibitive computational costs. Mixture-of-Experts (MoE) models\noffer an efficient alternative, increasing capacity without a proportional rise\nin compute requirements. However, training MoE models from scratch poses\nchallenges like overfitting and routing instability. We present an efficient\ntraining recipe leveraging pre-trained dense checkpoints, training an 8-Expert\nTop-2 MoE model from Llama 3-8B with less than 1 % of typical pre-training\ncompute. Our approach enhances downstream performance on academic benchmarks,\nachieving a \2% improvement in 0-shot accuracy on MMLU, while\nreaching a Model FLOPs Utilization (MFU) of \46.8% during training\nusing our framework. We also integrate online upcycling in NeMo for seamless\nuse of pre-trained weights, enabling cost-effective development of\nhigh-capacity MoE models.\n

Cited by

• Federated Fine-Tuning of Sparsely-Activated Large Language Models on Resource-Constrained Devices
• Scaling Fine-Grained MoE Beyond 50B Parameters: Empirical Evaluation and Practical Insights

Related

• LLaMA-MoE: Building Mixture-of-Experts from LLaMA with Continual Pre-training
• Scaling Fine-Grained MoE Beyond 50B Parameters: Empirical Evaluation and Practical Insights
• Scaling Laws for Upcycling Mixture-of-Experts Language Models
• Upcycling Large Language Models into Mixture of Experts
• Revisiting MoE and Dense Speed-Accuracy Comparisons for LLM Training