LLaMA Pro: Progressive LLaMA with Block Expansion

2024/01/04 by Chengyue Wu, Yukang Gan, Wu, Chengyue +14 · 1 voice · 15 citations
Computer Science · #Computation and Language (cs.CL) #FOS: Computer and information sciences #Multimodal Machine Learning Applications #Natural Language Processing Techniques #Topic Modeling #cs.CL

paper · pdf · doi:10.48550/arxiv.2401.02415

openalex publication_date 2024/01/04 · arxiv published 2024/01/04 · arxiv updated 2024/05/30 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Humans generally acquire new skills without compromising the old; however, the opposite holds for Large Language Models (LLMs), e.g., from LLaMA to CodeLLaMA. To this end, we propose a new post-pretraining method for LLMs with an expansion of Transformer blocks. We tune the expanded blocks using only new corpus, efficiently and effectively improving the model's knowledge without catastrophic forgetting. In this paper, we experiment on the corpus of code and math, yielding LLaMA Pro-8.3B, a versatile foundation model initialized from LLaMA2-7B, excelling in general tasks, programming, and mathematics. LLaMA Pro and its instruction-following counterpart (LLaMA Pro-Instruct) achieve advanced performance among various benchmarks, demonstrating superiority over existing open models in the LLaMA family and the immense potential of reasoning and addressing diverse tasks as an intelligent agent. Our findings provide valuable insights into integrating natural and programming languages, laying a solid foundation for developing advanced language agents that operate effectively in various environments.

Cited by

• MIDUS: Memory-Infused Depth Up-Scaling
• Merging without Forgetting: Continual Fusion of Task-Specific Models via Optimal Transport
• Deep Progressive Training: scaling up depth capacity of zero/one-layer models
• Motif 2 12.7B technical report
• SCALE: Upscaled Continual Learning of Large Language Models
• ADEPT: Continual Pretraining via Adaptive Expansion and Dynamic Decoupled Tuning
• Beyond Sunk Costs: Boosting LLM Pre-training Efficiency via Orthogonal Growth of Mixture-of-Experts
• Llama-GENBA-10B: A Trilingual Large Language Model for German, English and Bavarian
• Expanding Foundational Language Capabilities in Open-Source LLMs through a Korean Case Study
• Progressive Depth Up-scaling via Optimal Transport
• From Seeing to Experiencing: Scaling Navigation Foundation Models with Reinforcement Learning
• A Survey of Classification Tasks and Approaches for Legal Contracts
• Growing Transformers: Modular Composition and Layer-wise Expansion on a Frozen Substrate
• Dynamic Context-oriented Decomposition for Task-aware Low-rank Adaptation with Less Forgetting and Faster Convergence
• Flexible Realignment of Language Models

Discussions

• LLaMA Pro: Progressive LLaMA with Block Expansion [hn, 2 points, 1 comments]

Related

• Sheared LLaMA: Accelerating Language Model Pre-training via Structured Pruning
• Code Llama: Open Foundation Models for Code
• LLaMA-MoE: Building Mixture-of-Experts from LLaMA with Continual Pre-training
• LLaMA-MoE v2: Exploring Sparsity of LLaMA from Perspective of Mixture-of-Experts with Post-Training
• LlamaFactory: Unified Efficient Fine-Tuning of 100+ Language Models