FFTrainer: Fast Failover in Large-Language Model Training with Almost-Free State Management

2025/12/03 by Bohan Zhao, Y. Wang, Zhao, Bohan +12
Computer Science · #Distributed #FOS: Computer and information sciences #Natural Language Processing Techniques #Parallel #Parallel Computing and Optimization Techniques #Software System Performance and Reliability #and Cluster Computing (cs.DC)

paper · pdf · doi:10.48550/arxiv.2512.03644

openalex publication_date 2025/12/03 · openalex created_date 2025/12/05 · openalex updated_date 2026/07/28

Abstract

Recent developments in large language models (LLMs) have introduced new requirements for efficient and robust training. As LLM clusters scale, node failures, lengthy recoveries, and bulky checkpoints erode efficiency. Infrequent asynchronous checkpoints trigger costly rollbacks, yet higher frequencies add prohibitive overhead. To address these challenges, we propose FFTrainer, a system designed for robust LLM training. FFTrainer leverages surplus network capacity to quickly save and load states, thereby preventing rollbacks and accelerating recovery. Compared with prior checkpointing approaches, FFTrainer reduces recovery time by up to 98% and mitigates GPU utilization loss by up to 68% without hindering normal training.

Citations

• Nonuniform-Tensor-Parallelism: Mitigating GPU failure impact for Scaled-up LLM Training
• The Llama 3 Herd of Models
• ByteCheckpoint: A Unified Checkpointing System for Large Foundation Model Development
• Training Through Failure: Effects of Data Consistency in Parallel Machine Learning Training
• MegaScale: Scaling Large Language Model Training to More Than 10,000 GPUs
• TRANSOM: An Efficient Fault-Tolerant System for Training LLMs
• PyTorch FSDP: Experiences on Scaling Fully Sharded Data Parallel
• LLaMA: Open and Efficient Foundation Language Models
• Bamboo: Making Preemptible Instances Resilient for Affordable Training of Large DNNs
• Training language models to follow instructions with human feedback
• Efficient Large-Scale Language Model Training on GPU Clusters Using\n Megatron-LM
• ZeRO-Offload: Democratizing Billion-Scale Model Training
• CPR: Understanding and Improving Failure Tolerant Training for Deep Learning Recommendation with Partial Recovery
• PyTorch: An Imperative Style, High-Performance Deep Learning Library
• Checkmate: Breaking the Memory Wall with Optimal Tensor Rematerialization
• ZeRO: Memory Optimizations Toward Training Trillion Parameter Models
• Attention Is All You Need
• BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

Related

• FlashRecovery: Fast and Low-Cost Recovery from Failures for Large-Scale Training of LLMs
• Unicron: Economizing Self-Healing LLM Training at Scale
• GoCkpt: Gradient-Assisted Multi-Step overlapped Checkpointing for Efficient LLM Training
• TRANSOM: An Efficient Fault-Tolerant System for Training LLMs
• MegaScale: Scaling Large Language Model Training to More Than 10,000 GPUs