MeCeFO: Enhancing LLM Training Robustness via Fault-Tolerant Optimization

2025/10/18 by Rizhen Hu, Hu, Rizhen, Yutong He +8
Computer Science · #Advanced Neural Network Applications #Big Data and Digital Economy #Parallel Computing and Optimization Techniques

paper · pdf · doi:10.48550/arxiv.2510.16415

Abstract

As distributed optimization scales to meet the demands of Large Language Model (LLM) training, hardware failures become increasingly non-negligible. Existing fault-tolerant training methods often introduce significant computational or memory overhead, demanding additional resources. To address this challenge, we propose Memory- and Computation-efficient Fault-tolerant Optimization (MeCeFO), a novel algorithm that ensures robust training with minimal overhead. When a computing node fails, MeCeFO seamlessly transfers its training task to a neighboring node while employing memory- and computation-efficient algorithmic optimizations to minimize the extra workload imposed on the neighboring node handling both tasks. MeCeFO leverages three key algorithmic designs: (i) Skip-connection, which drops the multi-head attention (MHA) module during backpropagation for memory- and computation-efficient approximation; (ii) Recomputation, which reduces activation memory in feedforward networks (FFNs); and (iii) Low-rank gradient approximation, enabling efficient estimation of FFN weight matrix gradients. Theoretically, MeCeFO matches the convergence rate of conventional distributed training, with a rate of O(1/√(nT)), where n is the data parallelism size and T is the number of iterations. Empirically, MeCeFO maintains robust performance under high failure rates, incurring only a 4.18% drop in throughput, demonstrating 5.0× to 6.7× greater resilience than previous SOTA approaches. Codes are available at https://github.com/pkumelon/MeCeFO.

Citations

• CE-LoRA: Computation-Efficient LoRA Fine-Tuning for Language Models
• APOLLO: SGD-like Memory, AdamW-level Performance
• LDAdam: Adaptive Optimization from Low-Dimensional Gradient Statistics
• Subspace Optimization for Large Language Models with Convergence Guarantees
• Fira: Can We Achieve Full-rank Training of LLMs Under Low-rank Constraint?
• The Llama 3 Herd of Models
• FastPersist: Accelerating Model Checkpointing in Deep Learning
• SLTrain: a sparse plus low-rank approach for parameter and memory efficient pretraining
• VeLoRA: Memory Efficient Training using Rank-1 Sub-Token Projections
• DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model
• PiSSA: Principal Singular Values and Singular Vectors Adaptation of Large Language Models
• GaLore: Memory-Efficient LLM Training by Gradient Low-Rank Projection
• DropBP: Accelerating Fine-Tuning of Large Language Models by Dropping Backward Propagation
• DoRA: Weight-Decomposed Low-Rank Adaptation
• DeepSeek LLM: Scaling Open-Source Language Models with Longtermism
• TRANSOM: An Efficient Fault-Tolerant System for Training LLMs
• Llama 2: Open Foundation and Fine-Tuned Chat Models
• ReLoRA: High-Rank Training Through Low-Rank Updates
• Winner-Take-All Column Row Sampling for Memory Efficient Adaptation of Language Model
• AdaLoRA: Adaptive Budget Allocation for Parameter-Efficient Fine-Tuning
• LLaMA: Open and Efficient Foundation Language Models
• OPT: Open Pre-trained Transformer Language Models
• Bamboo: Making Preemptible Instances Resilient for Affordable Training of Large DNNs
• Training Compute-Optimal Large Language Models
• Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Model
• Varuna: Scalable, Low-cost Training of Massive Deep Learning Models
• TruthfulQA: Measuring How Models Mimic Human Falsehoods
• RoFormer: Enhanced Transformer with Rotary Position Embedding
• Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LM
• AdapterHub: A Framework for Adapting Transformers
• DAPPLE: A Pipelined Data Parallel Approach for Training Large Models
• Language Models are Few-Shot Learners
• PIQA: Reasoning about Physical Commonsense in Natural Language
• Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
• Root Mean Square Layer Normalization
• BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questions
• CodeNet: Training Large Scale Neural Networks in Presence of Soft-Errors
• Parameter-Efficient Transfer Learning for NLP
• Faster Neural Network Training with Approximate Tensor Operations
• GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding
• Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge
• Attention Is All You Need
• "Short-Dot": Computing Large Linear Transforms Distributedly Using Coded Short Dot Products
• Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer
• Layer Normalization
• Deep Residual Learning for Image Recognition
• Efficient Straggler Replication in Large-scale Parallel Computing
• Adam: A Method for Stochastic Optimization
• Failure Detection and Recovery in Hierarchical Network Using FTN Approach
• The tail at scale

Related

• Adacc: An Adaptive Framework Unifying Compression and Activation Recomputation for LLM Training
• AMSP: Reducing Communication Overhead of ZeRO for Efficient LLM Training
• GoCkpt: Gradient-Assisted Multi-Step overlapped Checkpointing for Efficient LLM Training
• ATTNChecker: Highly-Optimized Fault Tolerant Attention for Large Language Model Training
• BlockLLM: Memory-Efficient Adaptation of LLMs by Selecting and Optimizing the Right Coordinate Blocks